{ "claims": [ { "claim": "Published RD $t$-statistics in APSR, AJPS and JOP (2009–2018, 45 studies) bunch around and especially just above $1.96$; caliper tests find about three times as many main results just above as just below the threshold, which the paper interprets as consistent with selection for significant findings.", "claim_id": "P033-H1", "related_corrections": [ "P033-N002", "P033-N004" ], "tied_to": "Figure 1; Table 1" }, { "claim": "Estimated effect sizes are strongly associated with estimated standard errors, with estimates clustering around the $t=1.96$ line of the funnel plot; the paper suggests selection on statistical significance as an explanation.", "claim_id": "P033-H2", "related_corrections": [ "P033-E003", "P033-N002", "P033-N010" ], "tied_to": "Figure 4 (and Figure S4)" }, { "claim": "Bunching is stronger among studies using automated bandwidth selection than among studies using non-automated procedures, and a spike at $1.96$ survives re-estimation with a common automated bandwidth; the paper concludes that specification searching over bandwidths does not appear to be a main driver of the pattern.", "claim_id": "P033-H3", "related_corrections": [], "tied_to": "Figure 2; Figure 3(b); Supplement Tables S3, S4 and S5" }, { "claim": "The paper reports that reanalysis with rdrobust defaults leaves point estimates broadly similar, increases standard errors on average and moves $t$-statistics toward zero, which it interprets as evidence that published RD studies tend to overstate the accuracy of their findings. It reports similar results with RDHonest.", "claim_id": "P033-H4", "related_corrections": [ "P033-E001", "P033-E003", "P033-E006", "P033-N002", "P033-N006", "P033-N011" ], "tied_to": "Figures 3 and 4; Supplement Sections S7–S9, Figures S3–S9, Table S6" }, { "claim": "A retrospective power analysis of the 36 studies with data (64 analyses) reports that most are underpowered for all but large effects: 22% reach 80% power at $d=0.2$, 56% at $d=0.5$ and 70% at $d=0.8$.", "claim_id": "P033-H5", "related_corrections": [ "P033-E005", "P033-E010", "P033-N010", "P033-N011" ], "tied_to": "Table 2" }, { "claim": "The paper reports that studies using election data (68.9% of all studies, 72.2% of those with data) have considerably lower power than the others and show more bunching just above $1.96$; it suggests that scarce election data may help explain the contrast with economics.", "claim_id": "P033-H6", "related_corrections": [ "P033-E005", "P033-E010", "P033-N010", "P033-N011" ], "tied_to": "Table 3; Figure 5" }, { "claim": "The paper concludes that researchers tend to use inappropriate methods for inference, making standard errors artificially small, and that more than half of the sampled studies do not properly estimate the accuracy of their estimates.", "claim_id": "P033-H7", "related_corrections": [ "P033-N006" ], "tied_to": "Abstract, p. 1 (PDF p. 1); Results, p. 7 (PDF p. 7), paragraph on panel 3(c); Concluding remarks, p. 10 (PDF p. 10)" }, { "claim": "The paper contrasts its findings with Brodeur et al. (2020), who find little pathology for RD in economics. It reports ratios of tests just above to just below $1.96$ of about two in political science, attributed to Gerber and Malhotra (2008a), and about $1.1$ in economics, attributed to Brodeur et al. (2020).", "claim_id": "P033-H8", "related_corrections": [], "tied_to": "Introduction, p. 2 (PDF p. 2), right column; p. 5 (PDF p. 5), right column" }, { "claim": "Low power combined with selection pressure creates a risk that published RD effects are exaggerated or spurious; researchers should use modern bias-aware inference (Calonico et al. 2014; Armstrong and Kolesár 2020), restrict attention to adequately powered designs and pre-register, and journals should weigh power in publication decisions.", "claim_id": "P033-H9", "related_corrections": [ "P033-N004", "P033-N006", "P033-N011" ], "tied_to": "the analyses as a whole (Tables 1–3; Figures 1–5; Concluding remarks)" }, { "claim": "Weak identification assumptions for RD do not guarantee accurate estimation with limited data; local estimation involves bias and variance that inference must address.", "claim_id": "P033-H10", "related_corrections": [ "P033-E002", "P033-A001" ], "tied_to": "Section \"Why is it hard to estimate RD effects?\", pp. 3–4" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Scatter plot with horizontal axis 'Original P-Value' and vertical axis 'RDHonest P-Value', both from 0 to 1, with a red diagonal line and dashed lines at 0.05 on each axis; most points lie near the lower left corner. Red boxes mark the words 'RDHonest P-Value' in the title, the vertical-axis label 'RDHonest P-Value', and the words 'p-values derived using RDHonest' in the caption 'Figure S9: Original p-values vs. p-values derived using RDHonest'.", "caption": "Supplement, Figure S9, p. 18. The marked words, in the title, the vertical-axis label and the caption, call the plotted quantity a $p$-value of RDHonest.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E001.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T12", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T12" }, { "id": "P033-T13", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T13" } ], "headline": "The vertical axis of Figure S9 shows a normal-tail transform of the pseudo $t$-statistic built from one 95% bias-aware interval, which reproduces the 5% decision of that interval but is not in general a valid $p$-value at other levels.", "id": "P033-E001", "location": "Supplement, Figure S9, p. 18: title, vertical-axis label and caption", "pdf_page": 18, "printed_page": "18", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P033-T12", "P033-T13" ], "text": { "as_printed": "Title: ``Original versus \\hl{RDHonest P-Value}''. Subtitle: ``(Studies with Replication Data Reanalyzed with RDHonest)''. Horizontal axis: ``Original P-Value''. Vertical axis: ``\\hl{RDHonest P-Value}''.\n\nCaption: ``Figure S9: Original $p$-values vs. \\hl{$p$-values derived using \\texttt{RDHonest}}''.", "corrected": "Title: ``Original \\hl{p-values versus RDHonest pseudo p-values}''. Vertical axis: ``RDHonest \\hl{pseudo p-value}''.\n\nCaption: ``Figure S9: Original $p$-values versus \\hl{normal-tail transforms of RDHonest pseudo $t$-statistics. The vertical quantity is $p_*=2\\Phi(-z_{0.975}|\\widehat\\theta|/h)$, where $h$ is the reported 95\\% bias-aware confidence-interval half-length. The rule $p_*<0.05$ is exactly equivalent to this interval excluding zero. These transforms are not generally valid $p$-values at other significance levels; any 5\\% validity inherits the interval's coverage assumptions.}''\n\nThe plotted values are unchanged. A $p$-value that is valid at every level can be obtained by inverting a valid, nested family of bias-aware intervals, one for each level, under its assumptions on curvature and variance. In the Gaussian model $T\\sim N(\\theta+b,s^2)$ with known $s>0$ and known bound $|b|\\le B$, the value $\\Phi\\bigl((B-|T|)/s\\bigr)+\\Phi\\bigl((-B-|T|)/s\\bigr)$ is a valid $p$-value for $\\theta=0$. I do not give replacement values for the figure.", "evidence": "This correction rests on a derivation, an exact counterexample and a comparison with the replication files. I checked the Figure~S9 labels in the released figure and the caption in the supplement's extracted text; I did not have the typeset supplement page. Since $\\Phi$ is strictly increasing, I have for every $h>0$ that\n\\[\np_*<0.05\\iff 2\\Phi\\bigl(-z|\\widehat\\theta|/h\\bigr)<2\\Phi(-z)\\iff|\\widehat\\theta|>h\\iff 0\\notin[\\widehat\\theta-h,\\widehat\\theta+h].\n\\]\nFor the counterexample, let $T\\sim N(\\theta+b,1)$, where the bias $b$ satisfies $|b|\\le B$ for a known bound $B=1$. Let $H$ be the unique positive number such that\n\\[\n\\Phi(H-B)-\\Phi(-H-B)=\\frac{19}{20}.\n\\]\nThe interval $[T-H,T+H]$ covers $\\theta$ with probability at least $19/20$ for every $\\theta$ and every $|b|\\le B$, with equality at $b=\\pm B$. Applied to this interval, the transform is $p_*(T)=2\\Phi(-z|T|/H)$. Fix $\\theta=0$, $b=1$ and $\\alpha=1/5$, and set $q=\\Phi^{-1}(9/10)$ and $C=Hq/z$. Then $p_*\\le 1/5$ exactly when $|T|\\ge C$, and since $T\\sim N(1,1)$, I have\n\\[\n\\Pr(p_*\\le 1/5)=\\Phi(1-C)+\\Phi(-1-C).\n\\]\nIn exact rational arithmetic, I find that $2.646145548215300$, I have $f'(a)=\\varphi(a-z)-\\varphi(a+z)>0$, which implies that the power is strictly larger for a smaller standard error. I had the data of 45 of the 64 estimates. Fitting \\texttt{rdrobust} again for these 45 reproduces the saved coefficients and standard errors to a relative error of $4.3\\times10^{-15}$. The 180 saved powers of these estimates equal $f(|\\tau|/(s\\sqrt{\\rho}))$ to within $7\\times10^{-16}$, where $\\rho=1$ without clusters and $\\rho=G/(G_L+G_R)$ with clusters.\n\n\\begin{audittable}{lrrrr}{The four estimates: cluster counts and the ratio of the squared standard error used for power to $s^2$.}\nEstimate & $G$ & $G_L$ & $G_R$ & $G/(G_L+G_R)$ \\\\\nGerber and Hopkins (2011) & 57 & 23 & 46 & $19/23$ \\\\\nHall and Thompson (2018) & 185 & 98 & 101 & $185/199$ \\\\\nLopes da Fonseca (2017) & 278 & 202 & 226 & $139/214$ \\\\\nSances (2017) & 204 & 133 & 130 & $204/263$ \\\\\n\\end{audittable}\n\n\\begin{audittable}{lrrr}{The three threshold crossings: saved power and the power $f(\\tau/s)$ with the robust standard error.}\nEstimate and effect size & Saved & With $s$ & Threshold crossed \\\\\nLopes da Fonseca (2017), $d=0.5$ & $0.774$ & $0.589$ & $0.60$ \\\\\nLopes da Fonseca (2017), $d=0.8$ & $0.991$ & $0.938$ & $0.95$ \\\\\nSances (2017), $d=0.8$ & $0.663$ & $0.554$ & $0.60$ \\\\\n\\end{audittable}\n\nThe powers of Gerber and Hopkins and of Hall and Thompson are also too large, by at most $0.04$, and cross no threshold. The largest reduction among the four estimates is $0.185$. As exact fractions, the Table~2 cells change from $31/48$ to $89/144$, from $113/144$ to $109/144$ and from $31/48$ to $89/144$, and the Table~3 cells from $61/104$ to $57/104$, from $81/104$ to $77/104$ and from $61/104$ to $57/104$. All 36 printed cells are reproduced from the saved powers before the change.\n\nTwo limits apply. The source of the \\texttt{rdpower} package was not available, and I do not say which of its lines produce the factor. The data files of seven further clustered estimates, from Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al.\\ (2016), belong to the replication files, but I did not have them. Their powers were not recomputed, and further cells may change. Note that the estimate of Broockman and Ryan (2016) also has clusters on both sides ($G=380$, $G_L=379$, $G_R=380$), but its powers round to $1$ with either standard error. Its standard error is the subject of \\fid{P033-N011}.", "problem": "The paper says that the power calculation uses the robust standard error of \\texttt{rdrobust}. For four sharp clustered estimates, from Gerber and Hopkins (2011), Hall and Thompson (2018), Lopes da Fonseca (2017) and Sances (2017), the saved power does not match this description. Let $s$ be the robust standard error of the \\texttt{rdrobust} fit. Let $G$ be the number of distinct clusters, and let $G_L$ and $G_R$ be the numbers of clusters with observations to the left and to the right of the cutoff. All counts are taken after incomplete observations are removed. Each saved power equals the power under the normal approximation with the standard error\n\\[\ns\\,\\sqrt{G/(G_L+G_R)}\n\\]\nin place of $s$. If there exists a cluster with observations on both sides of the cutoff, I have $G0$ be the sample standard deviation of the outcome among control units, $\\widehat\\tau$ an estimate and $\\widehat{\\mathrm{SE}}>0$ its standard error. The figure plots\n\\[\nx=|\\widehat\\tau|/s_0,\\qquad y=\\widehat{\\mathrm{SE}}/s_0,\n\\]\nso that $x/y=|\\widehat\\tau|/\\widehat{\\mathrm{SE}}=|t|$. Let $z=z_{.975}$. Since $y>0$ and $z>0$, I have $|t|\\ge z$ if and only if $y\\le x/z$. The boundary is the line $y=x/z$ through the origin, with slope $1/z\\approx0.5102$, and the significant estimates lie to its right.\n\nFor the counterexample, let $z=1.96=49/25$. The point $(x,y)=(1/2,1/2)$ has $|t|=1<1.96$. At height $y=1/2$ the line of slope $49/25$ passes through $x=25/98<1/2$. The point lies to the right of that line and is not significant.\n\nThe plotting script of the replication files draws the line with slope \\texttt{1/qnorm(0.975)} and divides both coordinates by \\texttt{y0\\_sd}. The analysis script computes this divisor with \\texttt{sd()} over the observations below the cutoff. For Boas and Hidalgo (2011), these are 568 observations with 82 ones and 486 zeros. Their sample variance is $123/994\\approx0.124$, and the saved divisor agrees with $\\sqrt{123/994}\\approx0.3518$. I checked this one study only.", "problem": "The paragraph misstates two features of Figure~4. The figure and the code that draws it are right. First, the estimates and their standard errors are divided by the sample standard deviation of the outcome among observations below the cutoff, not by the sample variance. Second, the figure has the standardized estimate on the horizontal axis and the standardized standard error on the vertical axis. On these axes the line $|t|=1.96$ has slope $1/1.96$, the reciprocal of the printed value. A line of slope $1.96$, with the rule that estimates to its right are significant, would classify estimates with $|t|<1.96$ as significant.", "what_survives": "Figure~4, its red line, the estimates, the standard errors and every classification as significant are unchanged. The power analysis uses its own standard deviation, that of control units within the default bandwidth (p.~9), which this correction does not touch. Only the two terms in the description change." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E003", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of the paragraph that begins 'The results of the reanalysis are largely unchanged'. A red outline marks the words 'Section S8' in its second sentence.", "caption": "p. 8 (PDF p. 8), left column. The marked reference is “Section S8”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E006.png" }, { "alt": "Crop of a supplement page with the heading 'S3 Yearly number of published RD studies', Table S2 with total 45, Table S3 with total 46, and a four-line note. Red outlines mark 'Table' at the end of one line with 'A3' at the start of the next, then 'Table A3', then 'Table A2'.", "caption": "Online supplement, p. 8: Tables S2 and S3 and the note below them. The marked labels are “Table A3” (twice) and “Table A2”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E006-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The rdhonest results are in Section S9 of the supplement, not Section S8, and the note below Table S3 calls Tables S3 and S2 “Table A3” and “Table A2”.", "id": "P033-E006", "location": "p. 8 (PDF p. 8), left column, paragraph on the rdhonest procedure; online supplement, p. 8, note below Table S3", "pdf_page": 8, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Main text, p.~8: ``\\hl{Section S8} in the supplement reports the results using this alternative approach.''\n\nSupplement, p.~8: ``Therefore, we count this article twice in \\hl{Table A3}, once for each bandwidth selection, and so \\hl{Table A3} includes one more unit (46) than \\hl{Table A2} (45).''", "corrected": "Main text, p.~8: ``\\hl{Section S9} in the supplement reports the results using this alternative approach.''\n\nSupplement, p.~8: ``Therefore, we count this article twice in \\hl{Table S3}, once for each bandwidth selection, and so \\hl{Table S3} includes one more unit (46) than \\hl{Table S2} (45).''", "evidence": "The evidence is a check of the page for the main text. For the supplement I had the text of the official file and the page image shown above, not the PDF itself. The headings of Sections S8 and S9 and the titles and totals of Tables S2 and S3 are as stated. The table files of the replication files carry the same two tables.", "problem": "Two cross-references point to the wrong place. Section~S8 of the supplement is ``All studies with replication data''. The results of the rdhonest reanalysis are in Section~S9, ``Re-analysis using rdhonest package''. The reference to Section~S8 on p.~7, for the reanalysis of all 36 studies, is right. The supplement has no Table A2 or A3. The note stands below Tables S2 and S3 and explains their totals, 45 and 46.", "what_survives": "Correcting these labels changes no number or result." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E006", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of six lines of the right column of page 4. Red boxes mark the words 'between 2009 and' at the end of one line and '2018.' at the start of the next, in the sentence 'We collected all studies using an RD design published in the American Political Science Review, American Journal of Political Science, and Journal of Politics between 2009 and 2018.'", "caption": "Section “The state of RD studies in political science”, first sentence, p. 4 (PDF p. 4), right column. The marked words give the sample period, without saying how an article is dated.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E009.png" }, { "alt": "Crop of one entry of the reference list of the supplement: 'Schickler, E., Pearson, K., and Feinstein, B. D. (2010). Congressional parties and civil rights politics from 1933 to 1972. The Journal of Politics, 72(3), 672-689.' A red box marks the year '(2010)'.", "caption": "Supplement, reference list, p. 23. The supplement itself cites the study with the marked year 2010.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E009-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Tables S2 and S3 count the study of Schickler, Pearson and Feinstein in 2009, but it was published in 2010, and the 2009 and 2010 entries of four rows are each off by one.", "id": "P033-E009", "location": "Supplement, Section S3, Tables S2 and S3, p. 8; the sample period is stated on p. 4 (PDF p. 4) of the main text, right column", "pdf_page": null, "printed_page": "8", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Main text, p.~4: ``We collected all studies using an RD design published in the \\emph{American Political Science Review}, \\emph{American Journal of Political Science}, and \\emph{Journal of Politics} between 2009 and 2018.''\n\n\\begin{audittable}{llrrr}{Table~S2 (Yearly Published RD Articles) and Table~S3 (Bandwidth Selection in Sample of RD Articles), Supplement p.~8: the columns 2009, 2010 and Total, as printed. The columns 2011 to 2018 are not affected.}\nTable & Row & 2009 & 2010 & Total \\\\\nS2 & APSR & 1 & 0 & 19 \\\\\nS2 & AJPS & 0 & 0 & 10 \\\\\nS2 & JOP & \\hl{1} & \\hl{0} & 16 \\\\\nS2 & All & \\hl{2} & \\hl{0} & 45 \\\\\nS3 & Automated (e.g. CCT and IK) & 1 & 0 & 22 \\\\\nS3 & Non-automated & \\hl{1} & \\hl{0} & 20 \\\\\nS3 & Global Polynomial & 0 & 0 & 4 \\\\\nS3 & All & \\hl{2} & \\hl{0} & 46 \\\\\n\\end{audittable}", "corrected": "\\begin{audittable}{llrrr}{Tables~S2 and~S3: the columns 2009, 2010 and Total, corrected. The columns 2011 to 2018 are as printed.}\nTable & Row & 2009 & 2010 & Total \\\\\nS2 & APSR & 1 & 0 & 19 \\\\\nS2 & AJPS & 0 & 0 & 10 \\\\\nS2 & JOP & \\hl{0} & \\hl{1} & 16 \\\\\nS2 & All & \\hl{1} & \\hl{1} & 45 \\\\\nS3 & Automated (e.g. CCT and IK) & 1 & 0 & 22 \\\\\nS3 & Non-automated & \\hl{0} & \\hl{1} & 20 \\\\\nS3 & Global Polynomial & 0 & 0 & 4 \\\\\nS3 & All & \\hl{1} & \\hl{1} & 46 \\\\\n\\end{audittable}\n\nIf Cavaillé and Marshall is retained in 2018, Section~S3 should state the dating rule: \\hl{Articles are counted in the year of their first online publication. Under this rule Cavaillé and Marshall (2019) is counted in 2018.}", "evidence": "This correction rests on a recalculation from the replication files and on the publisher's dates. When the table code is run unchanged, it reproduces Tables~S2 and~S3. When only the year of this study is changed to 2010, the eight highlighted entries change and no others, and Tables~2, 3 and~S6, which the same script produces, do not change. The entries of Tables~S2 and~S3 are taken from the extracted text of the supplement and from the tables in the replication files, which agree; I did not have the typeset supplement pages.", "problem": "The two tables count Schickler, Pearson and Feinstein in 2009. Table~S1 and the reference list of the supplement cite the study as 2010, in \\emph{The Journal of Politics} 72(3), 672--689. According to the publisher, the article appeared online on 12 July 2010 and in the print issue of July 2010, and no date in 2009 is listed. The study therefore belongs in 2010 whether an article is dated by its first online publication or by its issue. In the replication code the year is read from the internal label of the article, \\texttt{schickler\\_etal\\_2009}. The study is in the JOP row of Table~S2, the Non-automated row of Table~S3 and both All rows. In each of these four rows the 2009 entry is one too high and the 2010 entry is one too low.\n\nA second study is counted in a year other than its cited year. Cavaillé and Marshall is cited as 2019, in \\emph{American Political Science Review} 113(1), and is counted among the seven APSR articles of 2018. It appeared online on 15 November 2018 and in the issue of February 2019. Its count in 2018, and its place in the period 2009--2018, are correct if articles are dated by first online publication. The paper does not state this rule.", "what_survives": "All row totals, the sample of 45 studies, and every estimate and weight are unchanged, since no analysis uses the year of publication. Only the eight annual counts change." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E009", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of four lines of the left column of page 4. A red box marks the word 'worse-case' in 'with the worse-case bias consistent with the observed data'. The next sentence begins 'The worst-case is taken with respect to a stipulated class of functions'.", "caption": "Paragraph beginning “The third approach”, p. 4 (PDF p. 4), left column. The marked word should read “worst-case”, as in the next sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008.png" }, { "alt": "Crop of five lines of the left column of page 11. A red box marks the words 'it as possible,' in the sentence 'However, insofar as it as possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.'", "caption": "Paragraph on preregistration, p. 11 (PDF p. 11), left column. The marked words should read “it is possible”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-b.png" }, { "alt": "Crop of the heading 'S6 Reproduction of originally reported estimates' and the two paragraphs under it. Red boxes mark the words 'This is section' at the start of the first paragraph and the words 'the manually feed' in the entry for de Benedictis-Kessner and Warshaw (2016).", "caption": "Supplement, Section S6, p. 12. The first marked phrase should read “In this section” and the second “then manually feed”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-c.png" }, { "alt": "Crop of the heading 'S9 Re-analysis using rdhonest package' and the paragraph under it. A red box marks the words 'under under' at the start of the third line, before 'second order Hölder class'.", "caption": "Supplement, Section S9, p. 16, first paragraph. The marked word is printed twice.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E008-d.png" } ], "formal_release": null, "formal_statements": [], "headline": "“worse-case” should read “worst-case”, “insofar as it as possible” should read “insofar as it is possible”, “This is section” should read “In this section”, “the manually feed” should read “then manually feed”, and one “under” in “under under” should be deleted.", "id": "P033-E008", "location": "p. 4 (PDF p. 4), left column; p. 11 (PDF p. 11), left column; Supplement, Section S6, p. 12, and Section S9, p. 16", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{itemize}\n\\item p.~4: ``The third approach is to adjust the inference procedure with the \\hl{worse-case} bias consistent with the observed data.''\n\\item p.~11: ``However, insofar as it \\hl{as} possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.''\n\\item Supplement, p.~12: ``\\hl{This is section} we document all failures to reproduce the authors' original estimates during our replication.''\n\\item Supplement, p.~12: ``When reproducing their estimates we select the bandwidth through the \\texttt{rdbwselect\\_2014} for backward compatibility purposes, and \\hl{the} manually feed this bandwidth into the \\texttt{rdrobust} package.''\n\\item Supplement, p.~16: ``\\ldots to estimate a bound on the second derivative for inference under \\hl{under} second order Hölder class \\ldots''\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item p.~4: ``The third approach is to adjust the inference procedure with the \\hl{worst-case} bias consistent with the observed data.''\n\\item p.~11: ``However, insofar as it \\hl{is} possible, having a full record of analyses that were intended would be helpful for the accumulation of knowledge.''\n\\item Supplement, p.~12: ``\\hl{In this section} we document all failures to reproduce the authors' original estimates during our replication.''\n\\item Supplement, p.~12: ``\\ldots and \\hl{then} manually feed this bandwidth into the \\texttt{rdrobust} package.''\n\\item Supplement, p.~16: ``\\ldots for inference \\hl{under} second order Hölder class \\ldots''\n\\end{itemize}", "evidence": "I checked the pages for the two phrases of the main text. The sentence after the first one prints ``The worst-case is taken with respect to a stipulated class of functions'', and p.~8 prints ``worst-case bias adjustment''. I compared the three supplement phrases in its extracted text with the marked images of those passages accompanying this entry; they agree. I did not have the typeset supplement pages for comparison.", "problem": "Each of the five phrases has a wrong, missing or repeated word. The meaning is clear in every case.", "what_survives": "Everything else is unaffected, since no statement of the paper depends on these words." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E008", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the section heading 'Why is it hard to estimate RD effects?' and the first nine lines of its opening paragraph. A red box marks the words 'RD design' in the first sentence, which begins 'The causal effect studied in the RD design is the difference'.", "caption": "Section “Why is it hard to estimate RD effects?”, first paragraph, p. 3 (PDF p. 3), left column. The marked words name the design without restricting it to the sharp case.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-A001.png" } ], "formal_release": null, "formal_statements": [], "headline": "The opening identification argument is the argument for a sharp RD design, and the paragraph does not say “sharp”.", "id": "P033-A001", "location": "Section “Why is it hard to estimate RD effects?”, first paragraph, p. 3 (PDF p. 3), left column", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "The causal effect studied in the \\hl{RD design} is the difference between the expected treated and control potential outcomes at a cutoff where treatment assignment changes in a discontinuous fashion. If the conditional expectation function of the outcome were known, this effect can be calculated without error under an assumption that the conditional expectation functions of the potential outcomes are continuous at the cut point. In other words, the RD effect is \\emph{identified} under a continuity assumption.", "corrected": "The causal effect studied in \\hl{a sharp} RD design is the difference between the expected treated and control potential outcomes at a cutoff where treatment assignment changes in a discontinuous fashion.\n\nA sentence on fuzzy designs may be added: \\hl{A fuzzy RD analysis uses the jump in the outcome divided by the jump in treatment receipt. For binary treatment receipt, under the additional continuity, exclusion, monotonicity and nonzero-first-stage assumptions, this ratio identifies an effect for compliers at the cutoff.}", "intended_meaning": "The paragraph states the identification argument for a sharp design, in which the running variable determines treatment. Let $X$ be the running variable and $c$ the cutoff, with the support of $X$ approaching $c$ from both sides. Set $D=\\mathbf{1}\\{X\\ge c\\}$ and $Y=DY(1)+(1-D)Y(0)$, and suppose that $\\mu_d(x)=\\mathbb{E}[Y(d)\\mid X=x]$ is continuous at $c$ for $d=0,1$. Then\n\\[\n\\lim_{x\\downarrow c}\\mathbb{E}[Y\\mid X=x]-\\lim_{x\\uparrow c}\\mathbb{E}[Y\\mid X=x]=\\mu_1(c)-\\mu_0(c)=\\mathbb{E}[Y(1)-Y(0)\\mid X=c],\n\\]\nwhich is the effect named in the first sentence. Read as this introductory case, the paragraph is correct.\n\nThe argument does not extend to a fuzzy design, in which only the probability of receiving treatment jumps at the cutoff. For example, suppose that half of the units are compliers with effect $2$ and half are never-takers. The two observed regression functions are the same whether the effect for the never-takers is $0$ or $10$, while the effect over all units is $1$ in the first case and $6$ in the second. The sample includes fuzzy designs, which are listed in Table~S1 of the supplement, and the replication code treats them separately from sharp designs. I found no formula or result of the paper that applies the argument outside the sharp case." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-A001", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of Table 2. A red outline marks the percent sign that precedes the words 'Effect size' in the heading of the first column.", "caption": "Table 2, p. 9 (PDF p. 9). The marked character is the percent sign in the heading of the first column.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E005.png" }, { "alt": "Crop of Table 3 with its two panels, 'Studies using election data' and 'Studies not using election data'. In each panel a red outline marks the percent sign that precedes the words 'Effect size'.", "caption": "Table 3, p. 10 (PDF p. 10), both panels. The marked characters are the percent signs in the two headings of the first column.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-E005-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The first column of Tables 2 and 3 gives effect sizes in units of Cohen’s $d$, and the percent sign in its heading does not belong there.", "id": "P033-E005", "location": "Table 2, p. 9 (PDF p. 9), and both panels of Table 3, p. 10 (PDF p. 10): heading of the first column", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "First-column heading of Table~2 and of both panels of Table~3: ``\\hl{\\%} Effect size'', above the rows 0.1, 0.2, 0.5 and 0.8.", "corrected": "``Effect size \\hl{(Cohen's $d$)}'', without the percent sign, in Table~2 and in both panels of Table~3. Deleting the percent sign alone is also correct.", "evidence": "The evidence is a check of the pages and of the replication files. The table files there carry the heading ``Effect size''. The analysis script multiplies each value of $d$ by the standard deviation of the outcome and does not convert it to a percentage.", "problem": "The rows of the first column are the four values of Cohen's $d$ defined on p.~9, in units of the standard deviation of the outcome. They are not percentages. The percent sign belongs only to the columns, which are the power thresholds 60, 80 and 95 under the heading ``Power (\\%)''. No entry of the tables needs rescaling.", "what_survives": "All entries of the tables, the power calculations and the comparisons drawn from them are unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-E005", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of five lines of the right column of page 4. A red box marks the words 'among all 45 studies in our sample.' in the sentence 'Figure 1 presents the distribution of t-statistics for the published findings among all 45 studies in our sample.'", "caption": "Paragraph beginning “Figure 1 presents the distribution”, p. 4 (PDF p. 4), right column. The marked words give the sample that the figure is said to show.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N002.png" }, { "alt": "Crop of Figure 1 with its caption: two histograms of t-statistics, panel (a) 'Main reported result' and panel (b) 'All primary results', each with a vertical line at 1.96, a horizontal axis with ticks at 0.98, 1.96, 2.94 and 3.92, and a vertical axis from 0 to 2. Red boxes mark the vertical-axis label 'Density' of panel (a) and the tick label '3.92' at the right end of its horizontal axis.", "caption": "Figure 1, p. 5 (PDF p. 5). The marked labels of panel (a) are the vertical axis, “Density”, and the last tick of the horizontal axis, $3.92$. The panel shows 37 of the 45 main results.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N002-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The histograms of Figures 1, 2, 3 and 5 leave out every $t$-statistic above $3.92$ and scale each density to the weight that remains, and each panel of Figure 4 leaves out 3 or 4 of the 39 estimates; the text and the captions do not say so.", "id": "P033-N002", "location": "p. 4 (PDF p. 4), right column, paragraph beginning “Figure 1 presents the distribution”; p. 7 (PDF p. 7); Figures 1, 2, 3, 4 and 5 (pp. 5, 6, 8 and 10)", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "p.~4: ``Figure~1 presents the distribution of $t$-statistics for the published findings \\hl{among all 45 studies in our sample}. The first panel presents the distribution of the main reported result of each article, meaning that each article contributes one $t$-statistic.''\n\np.~7: ``Figure~3 presents the distribution of $t$-statistics \\hl{among the 39 estimates} from the 25 studies included in the main replication exercise.''\n\np.~7: ``Figure~4 presents such a funnel plot \\hl{for the 39 estimates} in the replication study.''\n\nVertical axis of panels (a) of Figures~1, 2 and~3: ``\\hl{Density}''.", "corrected": "The following note should accompany Figures~1, 2, 3 and~5. \\hl{Each histogram shows the $t$-statistics in the interval from $0$ to $2\\Phi^{-1}(0.975)$, approximately $[0,3.92]$. The density is normalized to the weight in that interval, separately in each panel. The panels leave out 8 of 45, 12 of 80, 4 of 42, 8 of 38, 9 of 39, 10 of 39, 10 of 39, 7 of 47 and 5 of 33 statistics in Figures~1(a), 1(b), 2(a), 2(b), 3(a), 3(b), 3(c), 5(a) and 5(b), which is 17.8\\%, 16.1\\%, 13.6\\%, 18.5\\%, 21.0\\%, 26.0\\%, 29.0\\%, 13.7\\% and 21.4\\% of the weight.}\n\nThe following note should accompany Figure~4. \\hl{The panels show the estimates within the axis limits. Panels (a), (b) and (c) leave out 3, 3 and 4 of the 39 estimates.}\n\nIf the heights are instead to describe the full sample, each printed bar is multiplied by $V/W$, the reciprocal of the factor in the table above, and the weight outside the interval is reported.", "evidence": "This correction rests on a recalculation from the replication files and on a measurement of the published figure. The table lists what each panel leaves out, computed from the stored $t$-statistics and the weights of the paper.\n\n\\begin{audittable}{lrrrr}{Histogram panels: statistics that enter the code, statistics above the axis limit, share of weight left out, and the factor $W/V$ by which the printed bars exceed a density of the full sample.}\nPanel & Statistics & Left out & Share of weight left out & $W/V$ \\\\\n1(a) & 45 & 8 & $8/45=17.8\\%$ & $45/37=1.216$ \\\\\n1(b) & 80 & 12 & $29/180=16.1\\%$ & $180/151=1.192$ \\\\\n2(a) & 42 & 4 & $3/22=13.6\\%$ & $22/19=1.158$ \\\\\n2(b) & 38 & 8 & $17/92=18.5\\%$ & $92/75=1.227$ \\\\\n3(a) & 39 & 9 & $21/100=21.0\\%$ & $100/79=1.266$ \\\\\n3(b) & 39 & 10 & $13/50=26.0\\%$ & $50/37=1.351$ \\\\\n3(c) & 39 & 10 & $29/100=29.0\\%$ & $100/71=1.408$ \\\\\n5(a) & 47 & 7 & $17/124=13.7\\%$ & $124/107=1.159$ \\\\\n5(b) & 33 & 5 & $3/14=21.4\\%$ & $14/11=1.273$ \\\\\n\\end{audittable}\n\nThe bars of Figure~1(a) in the published PDF represent 37 results in total. A bar that holds one result has measured height $0.2482$. This agrees with $1/(37w)=0.2482$ and not with $1/(45w)=0.2041$. The figures in the replication files contain 36, 36 and 35 points for the three panels of Figure~4.", "problem": "The code for the histograms sets the horizontal axis to the interval from $0$ to $2\\Phi^{-1}(0.975)$, which is approximately $[0,3.92]$. A $t$-statistic above this limit enters no bin, and the density of each panel is then normalized to the weight that remains in the interval. Figure~1(a) shows 37 of the 45 main results, and each of its bars is higher by the factor $45/37$ than it would be in a density of all 45. Let $W$ be the total weight of the sample named in the text, let $V$ be the weight in the interval, and let $B_j$ be the weight in bin $j$, where each bin has width $w=\\Phi^{-1}(0.975)/18$. The printed height of bin $j$ is $B_j/(wV)$, and the height in a density of the full sample is $B_j/(wW)$. The printed heights are those of the distribution conditional on $t\\le 2\\Phi^{-1}(0.975)$.\n\nThe share of weight left out differs across panels. In Figure~3 it is 21\\%, 26\\% and 29\\% in panels (a), (b) and (c). The three panels that the text compares are conditioned on different sets. The funnel plots of Figure~4 have axis limits $0\\le x\\le 2$ and $0\\le y\\le 0.8$, and panels (a), (b) and (c) leave out 3, 3 and 4 of the 39 estimates. The main text, the captions and the text of the supplement state neither the omissions nor the normalization. The range of each axis is visible in the figures, and the counts 45, 80 and 39 given in the text are the correct sizes of the samples that enter the code.", "what_survives": "Within a panel, the order of the bars, the position of the spike near $1.96$ and the ratios between bars are unchanged, since every bar of the panel is rescaled by the same constant. Table~1 and Tables~S4 and~S5 use their designated main-estimate samples without these plotting limits, and their windows end below the axis limit. Tables~2 and~3 use all 64 analyses. In the full data the weighted share of $t$-statistics at or above $1.96$ is $0.77$, $0.63$ and $0.49$ for the samples of Figure~3(a), (b) and (c), so this decline in the share of significant results holds without the axis limit. The absolute scale of the densities and comparisons of bar heights between panels do change." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N002", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of seven lines of the right column of page 4. A red box marks the words 'reported p-values,' in the sentence 'For the small number of the studies where neither t-statistics nor standard errors were reported, we reconstruct t-statistics using reported p-values, confidence intervals and point estimates, or using replication code and data provided by the authors.'", "caption": "End of the paragraph on Figure 1, p. 4 (PDF p. 4), right column. The marked words name a source that the replication code does not use to reconstruct a $t$-statistic.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N003.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T14", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T14" }, { "id": "P033-T15", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T15" } ], "headline": "The replication code reconstructs no $t$-statistic from a reported $p$-value, and it computes a standard error from a confidence interval as the full width divided by twice the critical value, not from the upper limit and the point estimate as Section S4 says.", "id": "P033-N003", "location": "p. 4 (PDF p. 4), right column, last sentence of the paragraph beginning “Figure 1 presents the distribution”; Supplement, Section S4, pp. 9–10", "pdf_page": 4, "printed_page": "4", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P033-T14", "P033-T15" ], "text": { "as_printed": "Main text, p.~4: ``For the small number of the studies where neither $t$-statistics nor standard errors were reported, we reconstruct $t$-statistics using \\hl{reported $p$-values,} confidence intervals and point estimates, or using replication code and data provided by the authors.''\n\nSupplement, Section~S4, entry for Ariga (2015): ``The standard error estimate was imputed by (a) \\hl{subtracting the point estimate value from the upper value} of the reported 90\\% confidence interval, and then (b) dividing that value by the critical value for a 90\\% confidence interval (1.64).''\n\nSupplement, Section~S4, entries for Holbein (2016) and for Kalšnja and Titiunik (2017), as named in the extracted text: ``The standard error was not explicitly listed and so we imputed by using \\hl{the upper bound of a 95\\% CI and the point estimate} along with the critical value of 1.96.''", "corrected": "Main text, p.~4: ``For the small number of the studies where neither $t$-statistics nor standard errors were reported, we reconstruct $t$-statistics using \\del{reported $p$-values,} confidence intervals and point estimates, or using replication code and data provided by the authors.''\n\nSupplement, Section~S4, entry for Ariga (2015): the standard error is \\hl{the width $U-L$ of the reported 90\\% confidence interval $[L,U]$ divided by $2\\Phi^{-1}(0.95)$}.\n\nSupplement, Section~S4, entries for Holbein (2016) and for Klašnja and Titiunik (2017): the standard error is \\hl{the width $U-L$ of the reported 95\\% confidence interval $[L,U]$ divided by $2\\Phi^{-1}(0.975)$}.\n\nIn each case the $t$-statistic is the absolute point estimate divided by this standard error. Reported $p$-values are recorded separately, and a missing $p$-value is derived from the $t$-statistic by the normal approximation.", "evidence": "This correction rests on a derivation, an exact example and a recalculation from the replication files. I checked the Supplement Section~S4 quotations against its extracted text; I did not have the typeset supplement pages. Set $s_W=(U-L)/(2q)$ and $s_U=(U-b)/q$. Then\n\\[\n2q\\,(s_W-s_U)=(U-L)-2(U-b)=2b-U-L,\n\\]\nwhich implies that $s_W=s_U$ if and only if $b=(U+L)/2$. For Klašnja and Titiunik the recorded values are $b=-0.15$ and $[L,U]=[-0.213,-0.101]$. Then $U-L=0.112$ and $U-b=0.049$, and I obtain $s_W/s_U=8/7$ for every $q>0$. At the printed critical value $q=1.96$, I have $s_W=1/35$ and $s_U=1/40$, and the absolute $t$-statistics are $21/4=5.25$ and $6$. The replication code, which uses $q=\\Phi^{-1}(0.975)$, gives $5.2499$.\n\nOf the 81 rows of the raw metadata, 65 report a standard error, 1 reports a $t$-statistic, 2 use a 90\\% interval, 10 use a 95\\% interval and 3 use replication data. After every reported $p$-value is erased from the raw metadata, the cleaning code returns the same 81 standard errors and $t$-statistics.", "problem": "The description of how missing $t$-statistics were reconstructed differs from the replication code in two respects. First, the code fills a missing standard error from a reported $t$-statistic, from a 90\\% or 95\\% confidence interval, or from the replication data, and then computes every missing $t$-statistic as the absolute estimate divided by the standard error. No $t$-statistic is recovered from a reported $p$-value. Second, for a reported interval $[L,U]$ and point estimate $b$, the code computes the standard error as $(U-L)/(2q)$, where $q$ is $\\Phi^{-1}(0.95)$ for a 90\\% interval and $\\Phi^{-1}(0.975)$ for a 95\\% interval. Section~S4 instead describes the rule $(U-b)/q$ for Ariga, for Holbein, and for Klašnja and Titiunik. The two rules agree only when the interval is centered on the estimate. Of the six estimates to which Section~S4 applies this rule, four have a recorded interval that is not exactly centered. For three of them the two standard errors differ by less than 1\\%, and for Klašnja and Titiunik they differ by the factor $8/7$.", "what_survives": "Correcting the description changes no numerical result, and the cleaning code reproduces the cleaned data exactly. The reanalysis, its caliper tests and the power calculations do not use the imputed standard errors. If the rule of Section~S4 were applied instead, with its printed critical values, Tables~1 and~S4 would keep their counts and $p$-values. One estimate of Holbein (2016) would move by one histogram bin in Figures~1(b), 2(b) and~5(b), and six entries of Table~S6 would change." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N003", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of nine lines of the left column of page 5. Red boxes mark the words 'null hypothesis of no real pathologies' and, two lines below, 'equal success probability,' in the sentence 'We calculate the probability of observing an imbalance at least this extreme under the null hypothesis of no real pathologies in the RD literature by using a binomial distribution with equal success probability'.", "caption": "Paragraph beginning “Table 1 presents the results from the caliper tests”, p. 5 (PDF p. 5), left column. The first marked phrase is the null hypothesis that is named, and the second is the probability that the calculation uses.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N004.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T7", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T7" }, { "id": "P033-T8", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T8" } ], "headline": "The $p$-values of Table 1 are binomial tails under equal probability mass on the two sides of $1.96$ within each window, and this condition does not follow from the absence of selection on significance alone.", "id": "P033-N004", "location": "p. 5 (PDF p. 5), left column, paragraph beginning “Table 1 presents the results from the caliper tests”, and Table 1", "pdf_page": 5, "printed_page": "5", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P033-T7", "P033-T8" ], "text": { "as_printed": "We calculate the probability of observing an imbalance at least this extreme under the \\hl{null hypothesis of no real pathologies} in the RD literature by using a binomial distribution with \\hl{equal success probability}, capturing that we would expect a study to be equally likely to be just above and below the threshold under the null. [\\ldots] The $p$-values of the caliper tests range between 3\\% and 7\\%, depending on the window size, indicating that it is unlikely that we would observe this large of an imbalance in the $t$-statistics if there were no real pathologies in the RD literature.", "corrected": "The first of the two sentences quoted above is replaced by the following. ``\\hl{Conditional on the number of studies in each caliper, we calculate one-sided binomial tail probabilities under a working null of equal probability mass above and below the threshold. Interpreting these probabilities as evidence of publication bias requires an additional justification that the unselected distribution has approximately equal mass on the two sides, or a sensitivity analysis allowing unequal mass. Publication independence alone does not imply the working null.}'' The second sentence, on the range of 3\\% to 7\\%, is then a statement about this working null.\n\nIn full, the statement is as follows. Let $c=1.96$ and let $\\hl{00}$. Conditional on $n$ studies in the window $[c-h,c+h]$, the number $K$ above $c$ is binomial with parameters $n$ and $p_h=A_h/(A_h+B_h)$, and\n\\[\n\\Pr(K\\ge k\\mid n)=\\sum_{j=k}^{n}\\binom{n}{j}p_h^{\\,j}(1-p_h)^{n-j}.\n\\]\nThe entries of Table~1 are the values of this sum at $p_h=1/2$, which is the case \\hl{$A_h=B_h$}. If $f$ is nonincreasing on the window, then $p_h\\le 1/2$ and the one-sided test is conservative. If $f\\ge m>0$ and $f$ is $L$-Lipschitz on the window, then $|p_h-1/2|\\le Lh/(4m)$. Table~1 itself is unchanged.", "evidence": "This correction rests on exact counterexamples. Suppose that the studies are independent, that every study is published, and that the absolute $t$-statistic has density $f(x)=x/8$ on $[0,4]$. This density is smooth and positive on every window and has no jump at $1.96$. With $c=49/25$, a direct computation gives $A_h=(2ch+h^2)/16$ and $B_h=(2ch-h^2)/16$, so that $p_h=1/2+h/(4c)$. For $h=c/10$, $3c/20$ and $c/5$, I obtain $p_h=21/40$, $43/80$ and $11/20$. Given $n$ studies in the window, the number above $c$ is binomial with parameters $n$ and $p_h$. The table compares the upper tails at the counts of Table~1.\n\n\\begin{audittable}{lrrrr}{One-sided binomial tails at the counts of Table~1: with probability $1/2$, as printed, and with the probability $p_h$ of the density $f(x)=x/8$.}\nWindow & Below & Above & Tail with $1/2$ & Tail with $p_h$ \\\\\n10\\% & 3 & 9 & $299/4096=0.073$ & $0.100$ \\\\\n15\\% & 4 & 11 & $1941/32768=0.059$ & $0.102$ \\\\\n20\\% & 5 & 14 & $2083/65536=0.032$ & $0.078$ \\\\\n\\end{audittable}\n\nWith 19 studies in the 20\\% window, the test at the 5\\% level rejects exactly when at least 14 lie above $c$, and under this density it rejects with probability $0.0777$. In a second example each study has an unbiased normal estimate with standard error $1$ and true effect $7/2$, and every study is published. In the 20\\% window, $p_h$ then lies between $0.644747758104$ and $0.644747758105$, and the tail at 14 of 19 lies between $0.280413461677$ and $0.280413461678$. If instead every true effect is zero, then $p_h$ is approximately $0.3189$ and the one-sided test is conservative. I note that these examples are not estimates of $p_h$ for the 45 studies of the paper. For those studies I have established neither the size of the error nor its direction.", "problem": "The counts of Table~1 and its binomial tail probabilities are correct. The calculation takes each study in a window to lie above $1.96$ with probability $1/2$. Let $f$ be the density of the absolute $t$-statistic when publication does not depend on significance, set $c=1.96$, and let $h$ be the half-width of the window. Define\n\\[\nA_h=\\int_c^{c+h}f(x)\\,dx,\\qquad B_h=\\int_{c-h}^{c}f(x)\\,dx,\\qquad p_h=\\frac{A_h}{A_h+B_h}.\n\\]\nA study in the window lies above $c$ with probability $p_h$, and $p_h=1/2$ exactly when $A_h=B_h$. The null hypothesis that the 5\\% level plays no part in publication does not imply this equality at the half-widths $0.196$, $0.294$ and $0.392$ that the paper uses. If $f$ is differentiable at $c$ and $f(c)>0$, then\n\\[\np_h-\\frac12=\\frac{h\\,f'(c)}{4f(c)}+o(h),\n\\]\nwhich shows that the two counts are approximately equal as $h$ tends to zero, as the preceding paragraph of the paper says. No bound follows at a fixed $h$. The passage reads the tails of 3\\% to 7\\% as the probability of the imbalance when there is no selection, and that reading requires the further condition $A_h=B_h$, at least approximately. Equal mass on the two sides is the usual working assumption of the caliper test, which the paper takes from Gerber and Malhotra (2008a, 2008b).", "what_survives": "The counts, the imbalance of about three to one and the histograms are unchanged, and so are the entries of Table~1 when they are read as tails under equal mass on the two sides. The reanalysis and the power calculations do not use these probabilities. The reading of the 3\\% to 7\\% as the probability of the imbalance in the absence of selection does change. It now requires equal mass on the two sides, and the same holds for Tables~S4 and~S5 and for the discussion of bandwidth groups on p.~6." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N004", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the paragraph on panel 3(c). A red outline marks the words 'indicating that these studies have systematically overestimated the accuracy of their estimates.' at the end of the paragraph.", "caption": "Results, p. 7 (PDF p. 7), right column. The marked clause draws the inference from the shift in panel 3(c) to the accuracy of the original studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006.png" }, { "alt": "Crop of the last five lines of the abstract. A red outline marks the sentence 'However, researchers tend to use inappropriate methods for inference, rendering standard errors artificially small.'", "caption": "Abstract, p. 1 (PDF p. 1). The marked sentence states the diagnosis of the original studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-b.png" }, { "alt": "Crop of the first paragraph of the concluding remarks. A red outline marks the sentence 'More than half of the studies in our sample do not properly estimate the accuracy of their estimates, leading to the associated hypothesis tests and confidence intervals being misleading.'", "caption": "Concluding remarks, p. 10 (PDF p. 10), right column. The marked sentence gives the share of studies.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-c.png" }, { "alt": "Scatter plot on a logarithmic vertical axis. The 39 estimates are ordered by the ratio of the robust reanalysis standard error to the original standard error. Twelve open orange squares lie below the horizontal line at ratio 1: eleven between about 0.68 and 0.97, and one, labelled Broockman and Ryan, at 0.08. Twenty-seven blue circles lie above the line, rising from just above 1 to about 18.", "caption": "The comparison behind the three sentences, from the results saved in the replication files: for each of the 39 estimates of the main reanalysis, the robust standard error of the reanalysis divided by the original standard error. The lowest point is the estimate of Broockman and Ryan (2016), whose reanalysis standard error is corrected in P033-N011.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N006-se-ratios.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T1", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T1" }, { "id": "P033-T2", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T2" }, { "id": "P033-T16", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T16" } ], "headline": "The reanalysis shows that standardized standard errors become larger and absolute $t$-statistics smaller on average; it does not establish that the original studies used inappropriate inference, or that more than half of them did.", "id": "P033-N006", "location": "Results, p. 7 (PDF p. 7), right column, paragraph on panel 3(c); abstract, p. 1 (PDF p. 1); introduction, p. 2 (PDF p. 2); Concluding remarks, p. 10 (PDF p. 10)", "pdf_page": 7, "printed_page": "7", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": [ "P033-T1", "P033-T16", "P033-T2" ], "text": { "as_printed": "Abstract, p.~1: ``However, researchers tend to use \\hl{inappropriate methods for inference, rendering standard errors artificially small}.''\n\nResults, p.~7, on panel 3(c): ``Here, we see a clear shift to the left, \\hl{indicating that these studies have systematically overestimated the accuracy of their estimates}.''\n\nConcluding remarks, p.~10: ``\\hl{More than half of the studies in our sample do not properly estimate the accuracy of their estimates}, leading to the associated hypothesis tests and confidence intervals being misleading.''", "corrected": "Abstract, p.~1: ``However, \\hl{conventional RD inference that neglects nonnegligible smoothing bias can be misleading under the usual smoothness assumptions. A standardized reanalysis often produces larger estimated standard errors and smaller absolute test statistics than the original analyses. This pattern is consistent with understated uncertainty in the original findings, but it does not by itself establish that the original inference was inappropriate}.''\n\nResults, p.~7: ``Here, we see a clear shift to the left. \\hl{This pattern is consistent with understated uncertainty in the original studies, but changes in the estimator, the bandwidth and the precision of the bias estimate also affect it; the comparison alone does not establish that the original analyses were miscalibrated}.''\n\nConcluding remarks, p.~10: ``\\hl{In our standardized reanalysis of 25 studies, the robust standard error exceeds the original standard error for 27 of 39 estimates, and the absolute test statistic is smaller for 28 of 39. Standardized standard errors are larger and absolute test statistics smaller on average. This pattern is consistent with understated uncertainty in the original studies, but the comparison alone does not establish inappropriate original inference or the fraction of studies with misleading hypothesis tests and confidence intervals}.''\n\nThe sentence of the introduction that draws the same inference (p.~2, ``This indicates that the body of published RD studies tends to systematically overestimate the accuracy of their findings'') takes the same qualification: ``This \\hl{is consistent with a tendency of} the body of published RD studies to systematically overestimate the accuracy of their findings\\hl{, but does not establish it}.''", "evidence": "The evidence is a derivation with exact examples, a comparison under bounded curvature and undersmoothing, a recalculation of the counts from the results saved in the replication files, and a check of the pages.\n\nFor the example, let the running variable $X$ be uniform on $(-5,5)$ with a sharp cutoff at $0$, and let $Y_i(0)=2+X_i+\\varepsilon_i$ and $Y_i(1)=6+X_i+\\varepsilon_i$, where the errors $\\varepsilon_i$ are independent $N(0,1)$, independent of the scores, and their variance is known. The RD effect is $4$. I condition on eight observations with scores $\\pm1,\\pm2,\\pm3,\\pm4$ and compare three estimators. Each is the difference between a fitted intercept on the right and a fitted intercept on the left. The estimator $G$ fits a line to the four points on each side. The estimator $L$ is local linear with the triangular kernel and bandwidth $4$. The estimator $R$ subtracts from $L$ an estimate of its bias from a quadratic fit with the same kernel and bandwidth. With the points on one side ordered by their distance $1,2,3,4$ from the cutoff, the intercept weights of $G$, $L$ and $R$ are\n\\[\ng=(1,\\,1/2,\\,0,\\,-1/2),\\qquad l=(3/2,\\,0,\\,-1/2,\\,0),\\qquad r=(3,\\,-3,\\,1,\\,0).\n\\]\nEach vector of weights sums to $1$ and has first moment $0$, which implies that each estimator is unbiased for $4$. Since the errors are independent with variance $1$, I have\n\\[\n\\operatorname{Var}(G)=2\\sum_i g_i^2=3,\\qquad \\operatorname{Var}(L)=2\\sum_i l_i^2=5,\\qquad \\operatorname{Var}(R)=2\\sum_i r_i^2=38.\n\\]\nEach estimator is a linear combination of independent normal errors. I conclude that $(G-4)/\\sqrt3$, $(L-4)/\\sqrt5$ and $(R-4)/\\sqrt{38}$ are standard normal. For each of the three, the interval that extends $z_{.975}$ standard errors on either side of the estimate covers the effect with probability exactly $0.95$ at these specified affine conditional means and conditional on the eight scores. The standard error rises from $1.73$ to $2.24$ to $6.16$, with pointwise coverage unchanged. For the outcomes with errors $(1/10,-1/10,1/10,-1/10)$ on each side, all three estimates equal $4$, and the squared $t$-statistics are $16/3$, $16/5$ and $8/19$. Since $16/3>(1.96)^2=2401/625>16/5>8/19$, the first estimate is significant at $1.96$ and the two replacements are not.\n\n\\begin{audittable}{lrrrr}{The example: three estimators of an RD effect equal to 4, each with 95\\% coverage at the specified affine means; no coverage guarantee over a smoothness class is asserted.}\nEstimator & Variance & Standard error & Coverage & Squared $t$ \\\\\n$G$: linear fit on each side & $3$ & $1.73$ & $0.95$ & $16/3=5.33$ \\\\\n$L$: local linear, bandwidth $4$ & $5$ & $2.24$ & $0.95$ & $16/5=3.20$ \\\\\n$R$: $L$ with bias correction & $38$ & $6.16$ & $0.95$ & $8/19=0.42$ \\\\\nThreshold $(1.96)^2$ & & & & $3.84$ \\\\\n\\end{audittable}\n\nFor the opposite case, I define $\\psi(x)=(x^2-1)(x^2-4)(x^2-9)(x^2-16)/576$. This polynomial vanishes at the eight scores, and $\\psi(0)=1$. Let the mean of $Y(1)$ be $6+x+20\\psi(x)$ instead. The effect is now $24$, and the distribution of the eight outcomes is unchanged, so that the three estimates, their standard errors and the comparison between them are unchanged. The interval $G\\pm1.96\\sqrt3$ covers $24$ only if $|G-4|>16$. By Chebyshev's inequality, this event has probability at most $3/256$. The comparison is the same in the two cases. Coverage is exactly $95\\%$ at the first, affine pair of means and at most $1.2\\%$ at the second pair. These pointwise calculations show different actual coverage, not a coverage guarantee over a common smoothness class. Even a small quadratic perturbation, $\\mathbb{E}[Y(1)\\mid X=x]=6+x+ax^2$ with $a>0$, biases $G$ by $-5a$. At $a=1/10$ its conditional coverage is about $0.9404$, although the curvature is only $1/5$. Exact coverage at the affine means therefore cannot rebut the paper's smoothness-based criticism.\n\nA further example concerns covariates. Suppose that observed covariates $Z_i\\sim N(0,3)$, independent across observations and of the scores and errors, enter both potential outcomes with known coefficient $1$. Then the estimator $G$ applied to $Y-Z$ has variance $3$, and applied to $Y$ it has variance $12$. Both intervals have exact coverage at the stipulated covariate and affine data-generating process, and omitting the adjustment doubles the standard error. This does not establish validity for an original study under its maintained assumptions. The main call to \\texttt{rdrobust} in the replication files passes no covariates, and the original estimate of Gerber and Hopkins (2011) is coded there as adjusted for covariates.\n\nThere is also a comparison with valid inference over a smoothness class. Take scores $x_{+,j}=j/n$ and $x_{-,j}=-j/n$, $j=1,\\ldots,n$, independent $N(0,1)$ errors of known variance, and twice differentiable conditional means with $|m_d''(x)|\\le M$ for a fixed finite $M$. Put $m=\\lfloor n^{3/4}\\rfloor$ and $h=(m+1)/n$. Let $A$ be the local linear difference at the cutoff, using the first $m$ observations on each side with triangular weights $1-j/(m+1)$. Let $B$ subtract its estimated quadratic Taylor bias using the same bandwidth and kernel. The weighted normal equations make $B$ the difference of the local quadratic intercepts.\n\nFor $u=j/(m+1)$, the limiting intercept weight functions are\n\\[\nK_1(u)=(6-12u)(1-u),\\qquad K_2(u)=(12-60u+60u^2)(1-u).\n\\]\nTheir squared integrals are $24/5$ and $72/7$. Both fits reproduce constants and lines and have bounded sums of absolute intercept weights, so Taylor's theorem bounds each effect estimator's bias by a constant times $Mh^2$. Their variances are asymptotic to $2(24/5)/m$ and $2(72/7)/m$. Thus the absolute bias divided by its standard deviation is $O(h^2\\sqrt m)=O(n^{-1/8})$, uniformly over the stated mean class. The centered noise is exactly normal, so both conventional intervals have coverage tending uniformly to $0.95$. Yet $\\operatorname{Var}(B)/\\operatorname{Var}(A)\\to15/7$ and the ratio of standard errors tends to $\\sqrt{15/7}\\approx1.464$; for a fixed nonzero effect, the ratio of absolute test statistics for $B$ and $A$ tends in probability to $\\sqrt{7/15}<1$. This establishes the paper's undersmoothing exception without exact linearity. It does not reproduce any sampled study or the automatic bandwidths of \\texttt{rdrobust}.\n\n\\begin{audittable}{lrl}{Counts available in the paper, the supplement and the replication files. ``More than half'' of the sample requires 23 of 45 studies with inappropriate inference.}\nItem & Count & Basis \\\\\nStudies coded with the CCT bandwidth & 12 of 45 & a choice of bandwidth \\\\\nStudies coded with an IK or cross-validated bandwidth & 10 of 45 & a choice of bandwidth \\\\\nStudies coded with a non-automated bandwidth & 23 of 45 & a choice of bandwidth \\\\\nStudies in the main reanalysis & 25 of 45 & data available, not coded CCT \\\\\nEstimates with a larger robust standard error & 27 of 39 & estimates, not studies \\\\\nEstimates with a smaller absolute $t$-statistic & 28 of 39 & estimates, not studies \\\\\nStudies with a larger robust standard error & 16 of 25 & one main estimate per study \\\\\nEstimates above $1.96$: original, reanalysis & 30, 18 of 39 & two estimators compared \\\\\n\\end{audittable}\n\nThe 10 studies of the second row are 7 with an IK bandwidth and 3 with a cross-validated one, and the 23 studies of the third row include 4 with a global polynomial specification. Weighting each study equally, the mean standardized standard error rises from $0.188$ to $0.265$ and the mean absolute $t$-statistic falls from $5.03$ to $3.91$. The count of 16 studies holds for the main estimate of each study and for the mean ratio within a study, with different sets of 16. With the conventional standard error at the same bandwidth, the counts are 21 of 39 estimates and 12 of 25 studies.\n\nThe examples do not represent any study in the sample or the data-driven defaults of \\texttt{rdrobust}. I did not examine the original articles or their code. The pointwise affine calculations do not validate the original methods under the paper's smoothness assumptions, and the undersmoothing derivation does not show that the sampled studies used suitable bandwidths. The replacement comparison does not identify which original procedures failed their claimed coverage conditions or establish the asserted majority.", "problem": "The three sentences state a diagnosis of the original studies: their methods of inference were inappropriate, their standard errors were too small, and this holds for more than half of the 45 studies in the sample. The paper supports the diagnosis with a comparison. It estimates 39 effects from 25 studies again with the default settings of \\texttt{rdrobust}, and it compares the new standard errors and $t$-statistics with the published ones. The robust standard error of the reanalysis exceeds the original standard error for 27 of the 39 estimates, the absolute $t$-statistic is smaller for 28, and the number of estimates above $1.96$ falls from 30 to 18.\n\nThis comparison is between two estimators. To assess whether an original interval was appropriate for the RD argument, I need to assess its bias and sampling distribution, reported standard error and critical value under the maintained smoothness assumptions and any additional justified restrictions. Exact coverage at linear conditional means does not establish validity under those broader assumptions. The reanalysis changes the estimator and the bandwidth, adds a bias correction whose noise enters the standard error, and uses no covariates. These changes affect sampling variance. The finite example below has exact coverage at affine means and illustrates that variance comparison; it does not show validity across the smoothness class. Correct undersmoothing, which the paper explicitly accepts, can provide such validity without exact linearity, and a bias-corrected replacement may still have a larger standard error. The same comparison can also arise when the original interval almost never covers the effect. The comparison alone does not distinguish the two cases. This does not vindicate a method that lacks its claimed coverage guarantee under the maintained model; the paper need not establish actual curvature to criticize such a method. Its general warning about nonnegligible smoothing bias is sound. What still requires a connection to the original procedures is the empirical diagnosis and its stated prevalence.\n\nThe sentence on p.~10 needs a count. ``More than half'' of 45 studies means at least 23 studies with inappropriate inference. The paper does not classify the 45 studies by whether their inference is valid, and it does not say which count the sentence rests on. The relevant counts from the paper, the supplement and the replication files are listed in the second table below; none classifies inference validity. On pp.~3--4 the paper names three valid approaches: undersmoothing with conventional inference, bias correction with a standard error that accounts for it, and a worst-case adjustment for bias. If a study chose its bandwidth without an automated rule, or did not use \\texttt{rdrobust}, it does not follow that its inference is inappropriate. Note that exactly 23 studies are coded with a non-automated bandwidth. The paper does not present this count as the basis of the sentence, and by the criteria of pp.~3--4 it does not count inappropriate inference.\n\nThe diagnosis is not shown to be false. The wording of p.~7 is evidential (``indicating''), rather than a claim that every increase in a replacement standard error proves original undercoverage. The theory and reanalysis support concern about inference that neglects smoothing bias. The descriptions of the original methods and the replacement comparisons do not establish how many original studies failed the relevant bias-control conditions, particularly the asserted majority of 45. The more limited empirical conclusion is that the reanalysis is consistent with understated uncertainty.", "what_survives": "The descriptive comparison of the saved results stands: the counts 27 of 39 and 28 of 39, the fall from 30 to 18 estimates above $1.96$, and the larger standardized standard errors on average. The reading that this pattern is consistent with understated uncertainty survives. The general warning about nonnegligible smoothing bias on pp.~3--4 remains sound, as do the three valid routes to inference described there. The bunching of $t$-statistics, the caliper tests and the power analysis do not use the disputed prevalence claim. What changes is the empirical diagnosis in the abstract: it is qualified as a concern supported by the theory and the comparison, and the undocumented share ``more than half'' is withdrawn. The counts above use the saved values, including the erroneous Broockman and Ryan standard error addressed in \\fid{P033-N011}." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N006", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the paragraph that defines effect size by Cohen's d. A red outline marks the words 'the standard deviation of the outcome of control units'.", "caption": "Power analysis, p. 9 (PDF p. 9), left column. The marked words name the standard deviation used to standardize the effect.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010.png" }, { "alt": "Crop of Table 2. A red outline marks the cell 0.10 in the row for effect size 0.1 and the column for power 95.", "caption": "Table 2, p. 9 (PDF p. 9). The marked cell is the share of analyses with 95% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-b.png" }, { "alt": "Crop of the second panel of Table 3, 'Studies not using election data'. A red outline marks the cell 0.28 in the row for effect size 0.1 and the column for power 95.", "caption": "Table 3, p. 10 (PDF p. 10), panel “Studies not using election data”. The marked cell is the share with 95% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-c.png" }, { "alt": "Line chart. The horizontal axis is the standard deviation of the outcome on the scale of an individual probability of voting, from 0 to 0.74; the vertical axis is power at d = 0.1, from 0 to 1. The curve rises from 0.05 at zero to 0.897 at a standard deviation of 0.5, where a vertical line marks the largest population standard deviation of a binary outcome; the region beyond is shaded. Open circles on the curve mark standard deviations 0.343, 0.434 and 0.559, needed for 60, 80 and 95 percent power. A blue arrow at the top right marks the value as computed in the replication files: standard deviation 24.6 (voter counts per cell), power 1.", "caption": "Power at $d=0.1$ for Coppock and Green (2016) under the normal approximation of p. 9, as a function of the standard deviation that standardizes the effect, with the robust standard error held at its saved value (approximately $0.0155$). Open circles mark the standard deviations needed for 60%, 80% and 95% power.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N010-power-vs-sd.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T5", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T5" }, { "id": "P033-T6", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T6" }, { "id": "P033-T17", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17" }, { "id": "P033-T18", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T18" } ], "headline": "For Coppock and Green (2016), the estimate is an effect on the probability of voting but is standardized by the standard deviation of counts of voters, so that its “$d=0.1$” is not Cohen’s $d$ for the individual voting contrast; under the paper’s normal approximation with the saved standard error, the study cannot reach 95% power at $d=0.1$.", "id": "P033-N010", "location": "Power analysis, p. 9 (PDF p. 9), left column; Table 2, p. 9, row 0.1; Table 3, p. 10 (PDF p. 10), panel “Studies not using election data”, row 0.1", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P033-T17", "P033-T18", "P033-T5", "P033-T6" ], "text": { "as_printed": "p.~9: ``We measure effect size by Cohen's $d$, which is the treatment effect standardized by the standard deviation of the outcome (Cohen, 1988). We use \\hl{the standard deviation of the outcome of control units} within the default bandwidth in the rdrobust package for this standardization.''\n\nTable~2, p.~9, row 0.1 (power 60, 80, 95): 0.17, 0.15, \\hl{0.10}.\n\nTable~3, p.~10, studies not using election data, row 0.1: 0.40, 0.35, \\hl{0.28}.", "corrected": "p.~9: ``We measure effect size by Cohen's $d$, which is the treatment effect standardized by the standard deviation of the outcome (Cohen, 1988). We use the standard deviation of the outcome of control units within the default bandwidth in the rdrobust package for this standardization\\hl{, with the outcome and effect referring to the same treatment contrast. For Coppock and Green (2016), the intended contrast is an effect on an individual's probability of voting, so that the standard deviation needed is that of individual turnout among control units. The earlier code instead standardized an aggregate fuzzy-IV slope per prior voter by the standard deviation of outcome counts. That statistic is reproducible but does not give individual Cohen's $d$. Individual Cohen's $d$ cannot be computed from the replication data, and we report this study's standardized effect sizes as unavailable. Since the population standard deviation of a binary outcome is at most $1/2$, the study does not attain 95\\% power at $d=0.1$ under the normal approximation with the saved robust standard error}.''\n\nTable~2, row 0.1: 0.17, 0.15, \\hl{0.08}.\n\nTable~3, studies not using election data, row 0.1: 0.40, 0.35, \\hl{0.175}, which is about $0.18$.\n\nThe retained entries $0.17$ and $0.40$ are conditional on the study attaining 60\\% power at $d=0.1$; $0.15$ and $0.35$ are conditional on its attaining 80\\% power. Under the same approximation, the respective standard deviations of individual turnout required are approximately $0.343$ and $0.434$.", "evidence": "The evidence is an exact counterexample to interpreting the code-defined standardization as individual Cohen's $d$, a recalculation from the replication files, and a bound on power for that individual target.\n\nThe counterexample has two types of cell on each side of an eligibility cutoff. The two types are equally likely, and each cell has 100 individuals. On the eligible side, each cell has 50 voters in the earlier election, and the two types have 10 and 50 voters in the later one; five of the earlier voters in each cell are induced to vote later, so the later-voter counts without treatment would be 5 and 45. On the ineligible side, no one voted in the earlier election, and the two types have 10 and 40 voters in the later one. The fuzzy RD effect is $(30-25)/(50-0)=1/10$. Turnout without treatment is $1/4$ on each side, with variance $3/16$, and Cohen's $d$ satisfies $d^2=(1/10)^2/(3/16)=4/75$. The sample variance of the counts on the eligible side is $(10-30)^2+(50-30)^2=800$, and the standardization of the replication files gives $d^2=(1/10)^2/800=1/80000$. Doubling the population and the counts of every cell leaves the effect at $1/10$ and Cohen's $d^2$ at $4/75$. The variance of the counts becomes $3200$, and the second quantity becomes $1/320000$.\n\nFor the bound, a binary outcome with mean $p$ has variance $p-p^2\\le1/4$. Let $s$ be the saved robust standard error (approximately $0.0155$) and $z=z_{.975}$. Under the normal approximation, the power at $d=0.1$ with standard deviation $\\sigma$ is\n\\[\n\\Phi(\\mu-z)+\\Phi(-\\mu-z),\\qquad \\mu=0.1\\,\\sigma/s,\n\\]\nwhich increases in $\\mu\\ge0$. Substituting $\\sigma=1/2$, I obtain approximately $0.8973$. Solving for the standard deviations such that the power equals 60\\%, 80\\% and 95\\%, I find approximately $0.343$, $0.434$ and $0.559$. The last exceeds $1/2$. A sample standard deviation with divisor $N-1$ can exceed $1/2$ slightly. For a control group that contains all untreated later-election voters in the cells to the right of the cutoff within the bandwidth, at least 31,731 individuals, the bound is approximately $0.89727$.\n\nA fit of \\texttt{rdrobust} version 2.2 to the analysis file reproduces the saved coefficient and robust standard error to within $10^{-8}$, with bandwidth $105.66$ days; the original code used version 2.1.1. The local count standard deviation is $24.57954623$, so the code-defined target at $d=0.1$ is $2.45795462$. These reproduce the alternative actually computed, not individual Cohen's $d$. At the same bandwidths, doubling every count leaves the coefficient and its standard error unchanged. The study is one of the ten studies that do not use election data and has weight $1$. Changing its indicator for 95\\% power at $d=0.1$ from $1$ to $0$ changes the two cells as follows.\n\n\\begin{audittable}{lrr}{The two cells at effect size 0.1 and power 95\\%, with only this study changed.}\nCell & As printed & Corrected \\\\\nTable 2 & $5/48\\approx0.104$ & $11/144\\approx0.076$ \\\\\nTable 3, studies not using election data & $11/40=0.275$ & $7/40=0.175$ \\\\\n\\end{audittable}\n\nThe bound holds for the normal approximation with the saved standard error. It is not a statement about the exact power of the test in finite samples. I take the saved powers from the replication files; I have not independently executed \\texttt{rdpower}.", "problem": "The definition is right in general. For Coppock and Green, the code accurately computes an outcome-standardized aggregate fuzzy-IV coefficient per additional prior voter, a different comparison from Cohen's $d$ for individual turnout. In the analysis file for Coppock and Green (2016), each of the 140,659 rows is an aggregate: the outcome and the treatment are counts of voters, with maxima 294 and 552. The reanalysis is a fuzzy RD of one count on the other. Its estimate, $0.066$ with robust standard error $0.0155$, is a ratio of two discontinuities in counts. This is an aggregate outcome-count slope per prior voter; under the scientific interpretation in the paper's study metadata, the ratio identifies an effect on an individual's probability of voting. The code divides this effect by the standard deviation of the counts, $24.58$ within the bandwidth for the power analysis and $41.66$ below the cutoff for the funnel plots. The code's $d=0.1$ passes an alternative of $2.46$ to the fuzzy-RD slope calculation, and the saved power equals $1$ at all four effect sizes. That value is consistent with the code-defined target and saved standard error; the mismatch concerns its interpretation as individual Cohen's $d$. The divisor also depends on the size of the aggregates: at fixed bandwidths, doubling every count leaves the effect and its standard error unchanged and doubles the divisor.\n\nFor the intended individual voting contrast, the outcome is binary and its population standard deviation is at most $1/2$. An effect size of $d=0.1$ is then an effect of at most $0.05$, and under the normal approximation of the paper, with the saved robust standard error, the power at $d=0.1$ is at most about $0.8973$. Under this approximation, the study cannot attain 95\\% power at $d=0.1$, and one cell in each of Tables 2 and 3 changes. Whether the study attains 60\\% or 80\\% power at $d=0.1$ depends on the standard deviation of individual turnout, which the analysis file does not contain. The code also takes the observations to the left of the cutoff as the control units. In this file the treatment is concentrated on the left: within 200 days of the cutoff its mean is $100.7$ on the left and $0.42$ on the right. The left side is not a group of untreated individuals.", "what_survives": "The estimate itself, its standard error, its $t$-statistic and its $p$-value do not use the standardization, and the histograms of $t$-statistics and the caliper tests are unchanged. A common divisor cancels in the ratio of an estimate to its standard error, so that the study stays on the same side of the significance line in the funnel plots. What changes is its row of the power analysis and its position on the common scale of Figures 4, S4, S7 and S8. In panel (a) of Figure~4 it is plotted at about $(0.002,0.00005)$. The advantage of studies without election data at $d=0.1$ becomes smaller and keeps its sign. If this study also failed the 60\\% and 80\\% thresholds, the proportions would be $0.30$ and $0.25$, against $1/13\\approx0.08$ for election studies. Replacing the code-defined power of $1$ with power for the individual Cohen-$d$ target reinforces the conclusion that the studies are underpowered. The correction of \\fid{P033-N011} lowers the same cell of Table~2 for another study; each value here applies one correction alone." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N010", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the paragraph that begins 'Our primary reanalysis uses'. A red outline marks the words 'the default settings in the R package rdrobust', which run over two lines.", "caption": "Reanalysis method, p. 7 (PDF p. 7), left column. The marked words name the procedure used in the primary reanalysis.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-d.png" }, { "alt": "Crop of the paragraph that introduces Table 2. A red outline marks the figure 17 percent in the sentence 'For example, we see that only 17 percent of the analyses attain 60 percent power to detect a standardized effect of 0.1.'", "caption": "p. 9 (PDF p. 9), left column. The marked figure is the share of analyses with 60% power at effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011.png" }, { "alt": "Crop of Table 2. A red outline marks the three cells of the row for effect size 0.1: 0.17, 0.15 and 0.10.", "caption": "Table 2, p. 9 (PDF p. 9). The marked row is effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-b.png" }, { "alt": "Crop of the first panel of Table 3, 'Studies using election data'. A red outline marks the three cells of the row for effect size 0.1: 0.08, 0.08 and 0.04.", "caption": "Table 3, p. 10 (PDF p. 10), panel “Studies using election data”. The marked row is effect size 0.1.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-c.png" }, { "alt": "Two dot plots side by side with logarithmic horizontal axes and three rows: Broockman and Ryan (2016), and estimates (b) and (c) of Schickler et al. Left panel, robust standard error: for Broockman and Ryan the default value is 0.0022 (blue circle), against 0.079 with plug-in residuals at the saved bandwidth (orange square) and 0.033 with bandwidths selected again (open orange diamond); for the two Schickler estimates the three values lie close together between 0.09 and 0.16. Right panel, absolute robust t-statistic with a vertical line at 1.96: for Broockman and Ryan 23.04 by default against 0.65 and 1.83; for Schickler (b) about 2.0 by default against about 1.5 and 1.6; for Schickler (c) about 3.1 to 3.7.", "caption": "Robust standard error and absolute robust $t$-statistic of three clustered reanalyses, under the default variance (as saved in the replication files) and with plug-in residuals, at the saved bandwidths and with bandwidths selected again.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N011-se-by-variance.png" } ], "formal_release": null, "formal_statements": [ { "id": "P033-T3", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T3" }, { "id": "P033-T4", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T4" }, { "id": "P033-T17", "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17" } ], "headline": "With the default variance of rdrobust and clustering, a cell of at least four observations that share a value of the running variable inside one cluster adds nothing to the variance; the reanalysis standard error of Broockman and Ryan (2016) and its saved power of $1$ come from this cancellation, and correcting that study alone under the normal approximation turns the 17% of p. 9 into about 14%.", "id": "P033-N011", "location": "Table 2, p. 9 (PDF p. 9), row 0.1, and the sentence on 17% of the analyses; Table 3, p. 10 (PDF p. 10), panel “Studies using election data”, row 0.1; reanalysis method, p. 7 (PDF p. 7)", "pdf_page": 9, "printed_page": "9", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P033-T17", "P033-T3", "P033-T4" ], "text": { "as_printed": "p.~7: ``Our primary reanalysis uses \\hl{the default settings in the R package rdrobust} (Calonico et al., 2015).''\n\nTable~2, p.~9, row 0.1 (power 60, 80, 95): \\hl{0.17}, \\hl{0.15}, \\hl{0.10}.\n\np.~9: ``For example, we see that only \\hl{17\\%} of the analyses attain 60\\% power to detect a standardized effect of 0.1.''\n\nTable~3, p.~10, studies using election data, row 0.1: \\hl{0.08}, \\hl{0.08}, \\hl{0.04}.", "corrected": "p.~7: ``Our primary reanalysis uses the default settings in the R package rdrobust (Calonico et al., 2015)\\hl{, except for the variance in clustered designs in which observations in a cluster share values of the running variable. In such a design, the default nearest-neighbour residuals within a cell of four or more observations with the same value of the running variable sum to zero, and a cell of this kind inside one cluster contributes nothing to the cluster-robust variance. For these designs we replace those residuals, for example with regression plug-in residuals, retain the specified clusters and covariance when a cluster spans the cutoff, and propagate the change through bandwidth selection and power analysis}.''\n\nTable~2, row 0.1: \\hl{0.14}, \\hl{0.13} ($1/8$ exactly), \\hl{0.08}.\n\np.~9: ``For example, we see that only \\hl{14\\%} of the analyses attain 60\\% power to detect a standardized effect of 0.1.''\n\nTable~3, studies using election data, row 0.1: \\hl{0.04}, \\hl{0.04}, \\hl{0.00}.\n\nThese values correct the estimate of Broockman and Ryan (2016) alone, by the normal approximation with plug-in residuals, and hold every other saved power fixed. The cells at effect size 0.2 and power 95\\% become $0.15$ in Table~2 and $0.06$ in the election panel of Table~3. A full correction fits every clustered study again, including the three that were not checked here.", "evidence": "The evidence is a derivation, an exact counterexample, and a recalculation on the data of the replication files.\n\nLet $C$ be a cell of $m\\ge4$ observations with outcomes $y_i$ and sum $S$. The nearest-neighbour residual of observation $i$ is\n\\[\nr_i=\\sqrt{\\frac{m-1}{m}}\\Bigl(y_i-\\frac{S-y_i}{m-1}\\Bigr)=\\sqrt{\\frac{m}{m-1}}\\Bigl(y_i-\\frac{S}{m}\\Bigr),\n\\]\nwhich implies that $\\sum_{i\\in C}r_i=0$. The weight of an observation in the local polynomial fit depends only on its value of the running variable. The contribution of $C$ to the score of a cluster that contains it is this weight times $\\sum_{i\\in C}r_i$, which is zero. The same holds for the bias-corrected estimator.\n\nThe counterexample has ten independent clusters at $x=-5,\\dots,-1,1,\\dots,5$, with five observations each and cutoff $0$. The estimator is local linear with a quadratic fit for the bias, the triangular kernel and bandwidths $h=b=6$. Let $U_1,\\dots,U_{10}$ be independent and uniform on $\\{-1,1\\}$, and let every observation of cluster $g$ have the outcome $U_g$ under both treatments. The RD effect is $0$. The conventional estimator equals $\\sum_g a_gU_g$ with\n\\[\na=(1/5,\\,1/5,\\,0,\\,-2/5,\\,-1,\\,1,\\,2/5,\\,0,\\,-1/5,\\,-1/5),\n\\]\nand its variance is $\\sum_g a_g^2=62/25$. A direct computation gives the variance $10$ for the bias-corrected estimator. Every nearest-neighbour residual is $0$, and both estimated variances are $0$ in all $1024$ outcomes, while the two estimates are nonzero in $912$ and $968$ of them. If independent noise, uniform on $\\{-1,1\\}$, is added to every observation, then the covariance matrix of each cluster is positive definite. The true variances become $372/125$ and $12$, and the estimated variances stay $0$.\n\nMore clusters do not help. For $n\\ge3$, I place $n(n+1)$ clusters of five observations on each side at $x=\\pm j/(n(n+1))$ and set $h=b=1/n$, so that $n$ clusters on each side have positive weight. With outcome $U_g$ for every observation in cluster $g$, without the added observation noise, the variance of the conventional estimator is\n\\[\nV_n=\\frac{12(4n^2+5n-1)}{5n(n-1)(n+1)},\\qquad nV_n\\to48/5,\n\\]\nand the default estimate of it is $0$ for all $n$.\n\nOn the data of Broockman and Ryan, I find the same cancellation with exact rational arithmetic. In the union of the two bandwidth windows there are 43 cells on each side, each with at least four respondents, and all but one lie inside one district. The exception is the cell at $x=0.0013536513$ with 46 respondents from two districts, whose sums of deviations from the mean of the cell are $+19/46$ and $-19/46$. The sandwich formula applied to this cell alone reproduces both saved standard errors. A fit of \\texttt{rdrobust} with the default variance reproduces the saved coefficients and standard errors of the three estimates in the figure to within $4.8\\times10^{-16}$.\n\nFor the tables, I use the power under the normal approximation, $\\Phi(d\\,\\sigma_0/s-z_{.975})+\\Phi(-d\\,\\sigma_0/s-z_{.975})$, where $s$ is the robust standard error with plug-in residuals and $\\sigma_0$ the standard deviation of the outcome among control units within the bandwidth. At $d=0.1$ the power of the Broockman and Ryan estimate is $0.089$ at the saved bandwidths and $0.294$ with bandwidths selected again, below all three thresholds in both cases. The study has weight $1$ of 36, and of 26 among election studies.\n\n\\begin{audittable}{lrrr}{Cells that change when the Broockman and Ryan estimate alone is corrected, with every other saved power held fixed. The values are the same at the saved bandwidths and with bandwidths selected again.}\nCell (effect size, power) & As printed & Corrected & As a fraction \\\\\nTable 2 (0.1, 60\\%) & 0.17 & 0.139 & $5/36$ \\\\\nTable 2 (0.1, 80\\%) & 0.15 & 0.125 & $1/8$ \\\\\nTable 2 (0.1, 95\\%) & 0.10 & 0.076 & $11/144$ \\\\\nTable 2 (0.2, 95\\%) & 0.18 & 0.153 & $11/72$ \\\\\nTable 3, election data (0.1, 60\\%) & 0.08 & 0.038 & $1/26$ \\\\\nTable 3, election data (0.1, 80\\%) & 0.08 & 0.038 & $1/26$ \\\\\nTable 3, election data (0.1, 95\\%) & 0.04 & 0 & $0$ \\\\\nTable 3, election data (0.2, 95\\%) & 0.10 & 0.058 & $3/52$ \\\\\n\\end{audittable}\n\nAt the saved bandwidths, six further cells fall: the 60\\% and 80\\% cells at $d=0.2$ and the 95\\% cell at $d=0.5$, in Table~2 and in the election panel of Table~3. For Schickler et al., the robust standard error of estimate (b) at the saved bandwidths moves from $0.092$ to $0.122$ and its $t$-statistic from $2.00$ to $1.51$. The robust standard error of estimate (c) moves from $0.129$ to $0.153$.\n\nSeveral limits apply. The true sampling standard deviations are unknown, and the ratio of the two standard errors compares one estimator with another, not an estimator with the truth. Plug-in residuals remove the cancellation. I do not claim that the resulting intervals have correct coverage or that this is the only repair. Computing the conventional standard error by weighted least squares with errors clustered by district, I obtain $0.0630$ at the saved bandwidth. The \\texttt{rdpower} package was not available, and the powers above are the normal approximation. The paper used version 2.1.1 of \\texttt{rdrobust} and the recalculation used version 2.2, which reproduces the saved values.", "problem": "The reanalysis calls \\texttt{rdrobust} with the cluster variable of each study and the default variance estimator, which is built from nearest-neighbour residuals. The cancellation arises in the package's documented default combination of nearest-neighbour residuals and clustering, inherited through the paper's ordinary call; the paper did not independently write the cancelling variance formula. I call a set of observations with the same value of the running variable a cell. When a cell has $m\\ge4$ observations, the neighbours of each observation are the other $m-1$ members of the cell. The residual of an observation is then $\\sqrt{m/(m-1)}$ times the deviation of its outcome from the mean of the cell. For any values of the outcomes, these residuals sum to zero over the cell. The regression weights depend only on the running variable and are constant on the cell. A cell that lies inside one cluster then contributes zero to the score of that cluster. The uncertainty shared by the members of the cluster is left out of the variance. If every cell in the estimation and bias bandwidths has at least four observations and lies inside one cluster, the estimated variance is zero for the conventional and for the bias-corrected estimator.\n\nThe data of Broockman and Ryan (2016) nearly have this structure. The 29,413 survey respondents share 759 distinct values of the running variable, and the clusters are 380 districts. Inside the selected bandwidth there are 1,516 respondents in 36 cells, and 1,470 of them are in cells contained in one district. The only cell with a nonzero contribution is one of 46 respondents shared by two districts. The saved standard errors, $0.00211$ for the conventional estimate and $0.00222$ for the robust one, come from this cell alone. The reanalysis $t$-statistic of $23.0$ and the saved power of $1$ at every effect size follow from them. With regression plug-in residuals in place of the nearest-neighbour residuals and the same clusters, the robust standard error is $0.0790$ at the saved bandwidths, with $t=0.65$, and $0.0330$ when the bandwidths are selected again, with $t=1.83$.\n\nThe paper uses the default procedure as its reanalysis benchmark. Its statement that the cited methods represent progress is a general methodological judgment and is not refuted here. The application of this default as an uncertainty benchmark needs qualification for clustered designs in which observations in a cluster share values of the running variable. The paper should have checked whether the documented default was suitable for those designs. A larger number of clusters does not remove the problem. I established the cancellation for one study of 36. Two estimates of Schickler et al., with cells of two or three observations, are sensitive to the residuals to a lesser degree. Three other clustered studies, Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al.\\ (2016), were not checked, since I did not have their data files.", "what_survives": "The statistics reported by the original studies, the selection of studies, the bunching of the original $t$-statistics and the caliper tests on those original statistics do not use the reanalysis variance. Reanalyses without this cancellation are unaffected by this finding, and so is the rdhonest reanalysis, which is restricted to studies without clustering. The Broockman and Ryan replacements lower the apparent precision and power of that reanalysis, reinforcing the conclusions on low power and on larger reanalysis standard errors. Survival of these broad conclusions does not validate the affected default as an uncertainty benchmark. What changes is the Broockman and Ryan reanalysis: its standard errors, its bandwidth, its $t$-statistics, its place in panels (b) and (c) of Figures 3 and 4, and its power. With it change the first rows of Table~2 and of the election panel of Table~3, the quoted 17\\%, and the summaries of the supplement that use the same row. \\fid{P033-E010} concerns a different defect in the same tables and changes other cells. Under the stated normal approximation, \\fid{P033-N010} lowers the same 95\\% cell of Table~2 for another study, and each value here applies one correction alone." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N011", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of three entries of the reference list: Hahn, Todd and Klaauw (2001), Hall (2015), and Imbens and Kalyanaraman (2011). Red outlines mark 'Klaauw W' in the first entry and '(2011)' in the third.", "caption": "References, p. 12 (PDF p. 12), right column. The marked terms are the author name “Klaauw W” and the year “(2011)”.", "url": "https://pmaronow.github.io/llm-audit/works/P033/exhibits/P033-N013.png" } ], "formal_release": null, "formal_statements": [], "headline": "The third author of Hahn, Todd and Van der Klaauw (2001) is printed as “Klaauw W”, and the entry for Imbens and Kalyanaraman pairs the year 2011 with volume 79(3), which is an issue of 2012.", "id": "P033-N013", "location": "References, p. 12 (PDF p. 12), entries for Hahn et al. (2001) and Imbens and Kalyanaraman (2011); both are cited on p. 3", "pdf_page": 12, "printed_page": "12", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "Hahn J, Todd P and \\hl{Klaauw W} (2001) Identification and estimation of treatment effects with a regression-discontinuity design. \\emph{Econometrica} 69(1): 201--209.\n\nImbens G and Kalyanaraman K \\hl{(2011)} Optimal bandwidth choice for the regression discontinuity estimator. \\emph{The Review of Economic Studies} 79(3): 933--959.", "corrected": "Hahn J, Todd P and \\hl{Van der Klaauw W} (2001) Identification and estimation of treatment effects with a regression-discontinuity design. \\emph{Econometrica} 69(1): 201--209.\n\nImbens G and Kalyanaraman K \\hl{(2012)} Optimal bandwidth choice for the regression discontinuity estimator. \\emph{The Review of Economic Studies} 79(3): 933--959.\n\nThe year 2011 can be kept if the entry says that it is the year of advance online publication. The citation on p.~3 follows the year chosen.", "evidence": "The evidence is a check of the page and of the publishers' pages for the two articles. The byline of the \\emph{Econometrica} article reads ``Wilbert Van der Klaauw''. The page of the second article reads ``Volume 79, Issue 3, July 2012, Pages 933--959'', with the publication date 24 November 2011.", "problem": "The surname of the third author is Van der Klaauw, and the entry drops part of it. Volume 79, issue 3 of \\emph{The Review of Economic Studies} is the issue of July 2012. The article appeared online in advance on 24 November 2011. The year 2011 is a date of the article, but it does not match the volume and issue that the entry cites. Both entries still identify the works.", "what_survives": "No argument or result of the paper depends on either detail." }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-N013", "work": { "arxiv_id": "2109.14526", "doi": "10.1177/20531680231166457", "title": "On the reliability of published findings using the regression discontinuity design in political science", "url": "https://pmaronow.github.io/llm-audit/works/P033/" }, "work_id": "P033" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P033.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:0163aafdaca2931c10157794951c665288587422c52887a0f3a09835ac4bcdf2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P010/Wave1/Gap/GP010P033B1b.lean:190 states the claimed fixed-score M1 witness: both estimators are unbiased with the specified true variances and exact coverage, while the corrected standard error is larger and the observed significance comparison reverses. The product Gaussian law at P033/P033_Audit.lean:375 is exactly the conditional outcome law specified in the claim, so omitting latent random scores from this conjunction excludes no case. The Gram, weight, individual-intercept and one-side covariance displays are fixed finite auxiliary calculations, supplied separately by m1_exact_certificate at P033/P033_Audit.lean:399 and m1_wls_rows at :2365; the defined coefficient vectors are the stated WLS rows. Their packaging outside the credited conjunction does not materially weaken its pointwise counterexample component or change the estimators being compared.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.diagnostic_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P010/Wave1/Gap/GP010P033B1b.lean#L190", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:0163aafdaca2931c10157794951c665288587422c52887a0f3a09835ac4bcdf2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T1" }, "html": { "body": "

p. 7 (PDF p. 7), Results, paragraph beginning “Panel 3(c)”; Abstract, p. 1; Introduction, p. 2; Concluding remarks, p. 10. In the stipulated eight-score Gaussian sharp RD model, both weighted estimators are unbiased and have exact Gaussian inference, although the replacement has a larger standard error and loses realized significance. Includes both Gram and WLS certificates, every one-side variance and covariance, and the population and conditional-kernel construction. This is a pointwise witness at the specified affine means, with no claim of uniform coverage over a smoothness class.

", "headline": "Results, p. 7: ‘Here, we see a clear shift to the left, indicating that these studies have systematically overestimated the accuracy of their estimates’, with the Abstract (‘researchers tend to use inappropriate methods for inference, rendering standard errors artificially small’), read as the inference from a larger standard error and a smaller absolute t-statistic under the bias-corrected reanalysis to miscalibrated original inference; the witness is the adjudicating validator's construction and is not in the paper", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘Panel 3(c)’; Abstract, p. 1 (PDF p. 1); Introduction, p. 2 (PDF p. 2), left column; Concluding remarks, p. 10 (PDF p. 10)" }, "id": "P033-T1", "text": { "body": "p. 7 (PDF p. 7), Results, paragraph beginning “Panel 3(c)”; Abstract, p. 1; Introduction, p. 2; Concluding remarks, p. 10. In the stipulated eight-score Gaussian sharp RD model, both weighted estimators are unbiased and have exact Gaussian inference, although the replacement has a larger standard error and loses realized significance. Includes both Gram and WLS certificates, every one-side variance and covariance, and the population and conditional-kernel construction. This is a pointwise witness at the specified affine means, with no claim of uniform coverage over a smoothness class.", "headline": "Results, p. 7: ‘Here, we see a clear shift to the left, indicating that these studies have systematically overestimated the accuracy of their estimates’, with the Abstract (‘researchers tend to use inappropriate methods for inference, rendering standard errors artificially small’), read as the inference from a larger standard error and a smaller absolute t-statistic under the bias-corrected reanalysis to miscalibrated original inference; the witness is the adjudicating validator's construction and is not in the paper", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘Panel 3(c)’; Abstract, p. 1 (PDF p. 1); Introduction, p. 2 (PDF p. 2), left column; Concluding remarks, p. 10 (PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T1" }, { "declarations": [ { "claim_revision_id": "catalog:fc046d7678b65c2e0392f259adaa5a26ccc77d8a9bc67075a6079503e34d1813", "conditional_on": [], "credit_reason": "Conditional nonidentification is exactly the target: the full random-score population laws need not coincide. No empirical majority conclusion is exported or credited, and no hypothesis assumes the desired calibration result.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.diagnostic_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P010/Wave1/Gap/GP010P033B1b.lean#L427", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:fc046d7678b65c2e0392f259adaa5a26ccc77d8a9bc67075a6079503e34d1813", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T2" }, "html": { "body": "

Abstract, p. 1; Introduction, p. 2; Results, p. 7, paragraph beginning “Panel 3(c)”; Concluding remarks, p. 10 (printed and PDF pages agree). The affine model and its smooth quartic alternative induce the same conditional data law but different coverage. Includes the general Gaussian coverage formula, the polynomial and cutoff identities, bias and tail bounds, all study-count and nonidentification conclusions, and the square-integrable bias-adjustment variance identity. All conditional mean functions and the perturbation are C-infinity. The empirical counts in the replacement prose are expressly outside its formal claim.

", "headline": "The diagnosis of the Abstract (p. 1), of the Introduction (p. 2), of the Results paragraph on Panel 3(c) (p. 7) and of the Concluding remarks (p. 10, ‘More than half of the studies in our sample do not properly estimate the accuracy of their estimates’), as repaired by Sol: the reanalysis is consistent with understated original uncertainty but does not establish it, nor the fraction of studies affected (formal content: the comparison does not identify the calibration of the original inference)", "location": "Abstract, p. 1 (PDF p. 1); p. 2 (PDF p. 2), left column; p. 7 (PDF p. 7), Results, paragraph beginning ‘Panel 3(c)’; Concluding remarks, p. 10 (PDF p. 10)" }, "id": "P033-T2", "text": { "body": "Abstract, p. 1; Introduction, p. 2; Results, p. 7, paragraph beginning “Panel 3(c)”; Concluding remarks, p. 10 (printed and PDF pages agree). The affine model and its smooth quartic alternative induce the same conditional data law but different coverage. Includes the general Gaussian coverage formula, the polynomial and cutoff identities, bias and tail bounds, all study-count and nonidentification conclusions, and the square-integrable bias-adjustment variance identity. All conditional mean functions and the perturbation are C-infinity. The empirical counts in the replacement prose are expressly outside its formal claim.", "headline": "The diagnosis of the Abstract (p. 1), of the Introduction (p. 2), of the Results paragraph on Panel 3(c) (p. 7) and of the Concluding remarks (p. 10, ‘More than half of the studies in our sample do not properly estimate the accuracy of their estimates’), as repaired by Sol: the reanalysis is consistent with understated original uncertainty but does not establish it, nor the fraction of studies affected (formal content: the comparison does not identify the calibration of the original inference)", "location": "Abstract, p. 1 (PDF p. 1); p. 2 (PDF p. 2), left column; p. 7 (PDF p. 7), Results, paragraph beginning ‘Panel 3(c)’; Concluding remarks, p. 10 (PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T2" }, { "declarations": [ { "claim_revision_id": "catalog:6cefc2813bee113836f5d6abf081c316e9f409fac3c05422de2d48be0cc77485", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P033/P033_Audit.lean:2009 gives both positive true variances, identically zero NN cluster variances, the exact zero-estimate coverage counts, and the independent-record-noise extension with positive-definite within-cluster covariance requested by T3. Uniform averaging over the entire Boolean cube represents the stipulated independent uniform signs exactly, and counting estimates equal to zero is equivalent to coverage of the zero-width interval at effect zero. The explicit coefficient vectors at :642 and :646 agree with the claim's WLS estimators; the omitted matrix displays and row-identification calculations are fixed finite arithmetic stated separately in cluster_exact_certificate at :675 and cluster_wls_rows at :2379. No additional hypothesis, estimator restriction or nontrivial counterexample conclusion is lost.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.clustered_variance_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L1558", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6cefc2813bee113836f5d6abf081c316e9f409fac3c05422de2d48be0cc77485", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T3" }, "html": { "body": "

p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; Table 3, p. 10 (PDF p. 10); Figures 3 and 4, p. 8 (PDF p. 8). The ten-cluster sign model and its noisy extension have zero estimated variances despite positive true variances, with the stated WLS certificates and zero-width coverage counts.

", "headline": "The standard errors of the reanalysis for clustered designs (p. 7, ‘Our primary reanalysis uses the default settings in the R package rdrobust’; p. 9, ‘the bias adjustment and robust standard errors as implemented by rdrobust’), read as estimates of the sampling uncertainty of the clustered estimates; the witness is the adjudicating validator's construction and is not in the paper", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; Table 3, p. 10 (PDF p. 10); Figures 3 and 4, p. 8 (PDF p. 8)" }, "id": "P033-T3", "text": { "body": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; Table 3, p. 10 (PDF p. 10); Figures 3 and 4, p. 8 (PDF p. 8). The ten-cluster sign model and its noisy extension have zero estimated variances despite positive true variances, with the stated WLS certificates and zero-width coverage counts.", "headline": "The standard errors of the reanalysis for clustered designs (p. 7, ‘Our primary reanalysis uses the default settings in the R package rdrobust’; p. 9, ‘the bias adjustment and robust standard errors as implemented by rdrobust’), read as estimates of the sampling uncertainty of the clustered estimates; the witness is the adjudicating validator's construction and is not in the paper", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; Table 3, p. 10 (PDF p. 10); Figures 3 and 4, p. 8 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T3" }, { "declarations": [ { "claim_revision_id": "catalog:3022d676ed13e80292916c8841ab5c329928b3550205528a3bab9fa2d8eecc84", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P033/P033_Audit.lean:2396 includes the residual cancellation, shrinking-design variance and limit, noisy extension, general independent-cluster plug-in expectation, toy expectations and stipulated table arithmetic of T4. The normalized distance at :707 equals the claim's physical score divided by bandwidth, (j/[n(n+1)])/(1/n)=j/(n+1); suppressing clusters with zero kernel weight changes neither fit nor estimator law. The L2 premise at :2428 expresses the ordinary finite second moments already required by the target's common finite mean and variance, so it adds no substantive restriction. The only incorrect description is the target's asymptotic=false metadata: the claimed and formalized limit n V_n -> 48/5 appears explicitly at :2425.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.clustered_variance_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L1914", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:3022d676ed13e80292916c8841ab5c329928b3550205528a3bab9fa2d8eecc84", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T4" }, "html": { "body": "

Methodology, p. 7; Power analysis and Table 2, p. 9; Table 3, p. 10, election panel (printed and PDF pages agree). Within-cell nearest-neighbour residuals cancel in nested clusters. The full n(n+1)-cluster-per-side physical grid, with five records per cluster and h=b=1/n, has exactly n positive triangular weights per side and the stated normalized linear and same-bandwidth bias-corrected fits. The true sign-model variances are V_n and (6/5)V_n with record noise, n V_n tends to 48/5, and estimated nearest-neighbour variances are zero for every outcome array. Includes the independent finite-second-moment cluster plug-in identities, the ten-cluster examples, and the stipulated table-share arithmetic.

", "headline": "The use of the default rdrobust variance in the clustered reanalyses (pp. 7 and 9) and the entries of Tables 2 and 3 that rest on it, as repaired by Sol: the default nearest-neighbour residuals remove the uncertainty of running-variable cells nested in a cluster (formal content: the cancellation identity, its persistence with many clusters, a variance construction that keeps the cluster component, and the arithmetic of the affected table cells)", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Table 2 and the sentence ‘only 17% of the analyses attain 60% power’; Table 3, p. 10 (PDF p. 10), first panel" }, "id": "P033-T4", "text": { "body": "Methodology, p. 7; Power analysis and Table 2, p. 9; Table 3, p. 10, election panel (printed and PDF pages agree). Within-cell nearest-neighbour residuals cancel in nested clusters. The full n(n+1)-cluster-per-side physical grid, with five records per cluster and h=b=1/n, has exactly n positive triangular weights per side and the stated normalized linear and same-bandwidth bias-corrected fits. The true sign-model variances are V_n and (6/5)V_n with record noise, n V_n tends to 48/5, and estimated nearest-neighbour variances are zero for every outcome array. Includes the independent finite-second-moment cluster plug-in identities, the ten-cluster examples, and the stipulated table-share arithmetic.", "headline": "The use of the default rdrobust variance in the clustered reanalyses (pp. 7 and 9) and the entries of Tables 2 and 3 that rest on it, as repaired by Sol: the default nearest-neighbour residuals remove the uncertainty of running-variable cells nested in a cluster (formal content: the cancellation identity, its persistence with many clusters, a variance construction that keeps the cluster component, and the arithmetic of the affected table cells)", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Table 2 and the sentence ‘only 17% of the analyses attain 60% power’; Table 3, p. 10 (PDF p. 10), first panel" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T4" }, { "declarations": [ { "claim_revision_id": "catalog:b4d8db1e29e10e1bfc75b13db26fc3269b8c1a72de3e1bdc1c5747ca15df57dc", "conditional_on": [], "credit_reason": "The credited component compares individual-outcome Cohen’s d with aggregate count standardization; it does not refute the validity of an aggregate outcome-standardized fuzzy-IV target or establish an error in its power arithmetic.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.count_standardization_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L635", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b4d8db1e29e10e1bfc75b13db26fc3269b8c1a72de3e1bdc1c5747ca15df57dc", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T5" }, "html": { "body": "

Power analysis and Table 2, p. 9, row 0.1, column 95; Table 3, p. 10, second panel, row 0.1, column 95; Figure 4, p. 8. The binary individual-outcome fuzzy RD witness has individual squared Cohen effect 4/75 but count-standardized squared effect 1/80000. Doubling all cell counts preserves the Wald effect and individual standardization, while dividing the count-standardized squared effect by four. In general positive count scaling preserves the Wald ratio, scales the sample standard deviation, and divides the count-standardized effect by the scale. Wald and standardization denominators are nonzero, and sample standard deviations use at least two observations.

", "headline": "Power analysis, p. 9: ‘We measure effect size by Cohen’s d, which is the treatment effect standardized by the standard deviation of the outcome’, as implemented for Coppock and Green (2016), where an effect on an individual probability is divided by the standard deviation of cell-level voter counts; the witness is the validator's construction and is not in the paper", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), second panel (row 0.1, column 95); Figure 4, p. 8 (PDF p. 8)" }, "id": "P033-T5", "text": { "body": "Power analysis and Table 2, p. 9, row 0.1, column 95; Table 3, p. 10, second panel, row 0.1, column 95; Figure 4, p. 8. The binary individual-outcome fuzzy RD witness has individual squared Cohen effect 4/75 but count-standardized squared effect 1/80000. Doubling all cell counts preserves the Wald effect and individual standardization, while dividing the count-standardized squared effect by four. In general positive count scaling preserves the Wald ratio, scales the sample standard deviation, and divides the count-standardized effect by the scale. Wald and standardization denominators are nonzero, and sample standard deviations use at least two observations.", "headline": "Power analysis, p. 9: ‘We measure effect size by Cohen’s d, which is the treatment effect standardized by the standard deviation of the outcome’, as implemented for Coppock and Green (2016), where an effect on an individual probability is divided by the standard deviation of cell-level voter counts; the witness is the validator's construction and is not in the paper", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), second panel (row 0.1, column 95); Figure 4, p. 8 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T5" }, { "declarations": [ { "claim_revision_id": "catalog:19c9692cc5da16855534d59a555f900e5e262569bb8ab03051484225dc658042", "conditional_on": [], "credit_reason": "The mathematical target already fixes the individual-binary interpretation and stored-SE reference calculation; empirical standard deviations, fitted-data provenance and new table classifications are not proved by this row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.binary_standardization_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/Wave1/Gap/GLP010P033P033B2.lean#L696", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:19c9692cc5da16855534d59a555f900e5e262569bb8ab03051484225dc658042", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T6" }, "html": { "body": "

p. 9 (PDF p. 9), Power analysis and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), ‘Studies not using election data’ (row 0.1, column 95). Binary outcomes bound the standard deviation and the specified stored-SE normal power below 95%, with the stated derivative, root, sample-size, table and ratio conclusions.

", "headline": "The Cohen's d standardization of p. 9 as applied to Coppock and Green (2016), and the cells of Tables 2 and 3 that depend on it, as repaired by Sol: an effect on a binary individual outcome is standardized by a standard deviation that is at most 1/2, so the study cannot reach 95% power at d = 0.1 (formal content: the bound on a binary standard deviation, the power bound, the one-row table arithmetic, and the invariance of the t-ratio)", "location": "p. 9 (PDF p. 9), Power analysis and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), ‘Studies not using election data’ (row 0.1, column 95)" }, "id": "P033-T6", "text": { "body": "p. 9 (PDF p. 9), Power analysis and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), ‘Studies not using election data’ (row 0.1, column 95). Binary outcomes bound the standard deviation and the specified stored-SE normal power below 95%, with the stated derivative, root, sample-size, table and ratio conclusions.", "headline": "The Cohen's d standardization of p. 9 as applied to Coppock and Green (2016), and the cells of Tables 2 and 3 that depend on it, as repaired by Sol: an effect on a binary individual outcome is standardized by a standard deviation that is at most 1/2, so the study cannot reach 95% power at d = 0.1 (formal content: the bound on a binary standard deviation, the power bound, the one-row table arithmetic, and the invariance of the t-ratio)", "location": "p. 9 (PDF p. 9), Power analysis and Table 2 (row 0.1, column 95); Table 3, p. 10 (PDF p. 10), ‘Studies not using election data’ (row 0.1, column 95)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T6" }, { "declarations": [ { "claim_revision_id": "catalog:a9294b70cc502188faf439310194d376db112d2031bb753fb9f1422aaee6f52c", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P033/Wave1/Gap/GLP010P033P033B2.lean:1467 supplies the linear-density, polynomial-family and exactly valid Gaussian witnesses, including the conditional linear-model rejection calculation and folded-normal CDF and tail values. Its positive-probability guard at :1488 excludes no intended linear-model case: the 20% window has probability 2401/12500 strictly between zero and one, making P(N=19)>0 for every M>=19. Observing all iid draws represents always publishing; local smoothness loses no endpoint case because every allowed closed window lies strictly inside (0,4), and the folded-normal binomial-tail representation has the claimed conditional-count interpretation. The target's asymptotic=false metadata is incorrect because the polynomial-family limit as m tends to infinity is stated at :1507.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.caliper_counterexamples_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/Wave1/Gap/GLP010P033P033B2.lean#L1466", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:a9294b70cc502188faf439310194d376db112d2031bb753fb9f1422aaee6f52c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T7" }, "html": { "body": "

p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1. Selection-free linear, polynomial and Gaussian study populations have unequal caliper-side probabilities and the stated excess rejection tails.

", "headline": "p. 5: ‘a binomial distribution with equal success probability, capturing that we would expect a study to be equally likely to be just above and below the threshold under the null’, read as the claim that when the 1.96 threshold is irrelevant to publication a study inside a caliper is above the threshold with probability one half; the witnesses are the validators' constructions and are not in the paper", "location": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1" }, "id": "P033-T7", "text": { "body": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1. Selection-free linear, polynomial and Gaussian study populations have unequal caliper-side probabilities and the stated excess rejection tails.", "headline": "p. 5: ‘a binomial distribution with equal success probability, capturing that we would expect a study to be equally likely to be just above and below the threshold under the null’, read as the claim that when the 1.96 threshold is irrelevant to publication a study inside a caliper is above the threshold with probability one half; the witnesses are the validators' constructions and are not in the paper", "location": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T7" }, { "declarations": [ { "claim_revision_id": "catalog:871fc83180c3523e9414a8146b9372dfbb426d99c89fb5250cdb125ddf599551", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P033/Wave1/Gap/GLP010P033P033B3.lean:699 matches T8's corrected iid nonnegative-density model and includes the conditional binomial law, equal-mass criterion, monotonicity, finite-width bound, shrinking-window results and benchmark tails. The j<=n and k<=n guards lose only trivial zero cases: by the count definitions at P033/P033_Audit.lean:1015 and :1020, K<=N pointwise, while binomial masses and upper tails beyond n vanish. Including interval endpoints changes no probability under a density, and the right-hand filter at :732 is precisely h decreasing to zero. The target's asymptotic=false metadata should be corrected because the limit and little-o expansion are explicitly concluded at :731-735.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.caliper_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/Wave1/Gap/GLP010P033P033B3.lean#L701", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:871fc83180c3523e9414a8146b9372dfbb426d99c89fb5250cdb125ddf599551", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T8" }, "html": { "body": "

p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1. For the specified normalized density, caliper counts obey the stated conditional binomial laws, mass criteria, monotonicity, finite-width bounds and small-window limits.

", "headline": "The null of the caliper test (p. 5) and the reading of the p-values of Table 1, as repaired by Sol: the binomial tails are computed under a working null of equal probability mass on the two sides of 1.96 inside each caliper, which the absence of selection does not imply (formal content: the exact conditional law of the count, and conditions under which the benchmark is exact, conservative, or approximately right)", "location": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1" }, "id": "P033-T8", "text": { "body": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1. For the specified normalized density, caliper counts obey the stated conditional binomial laws, mass criteria, monotonicity, finite-width bounds and small-window limits.", "headline": "The null of the caliper test (p. 5) and the reading of the p-values of Table 1, as repaired by Sol: the binomial tails are computed under a working null of equal probability mass on the two sides of 1.96 inside each caliper, which the absence of selection does not imply (formal content: the exact conditional law of the count, and conditions under which the benchmark is exact, conservative, or approximately right)", "location": "p. 5 (PDF p. 5), left column and the continuation under Table 1; Table 1" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T8" }, { "declarations": [ { "claim_revision_id": "catalog:80c0d26701f51b4a3e44253129884f1db54fc2f4406ea2462924ee3c115e5e86", "conditional_on": [], "credit_reason": "The exact Gaussian law is the explicit reference model for the normal approximation in T9(i), not an additional finite-sample assumption placed on the published empirical procedure. The target supplies saved-power matches and threshold crossings externally; no separate common-SD invariance proposition is claimed by the original row description supplied in this packet.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.native_power_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P010/Wave1/Gap/GP010P033B2.lean#L225", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:80c0d26701f51b4a3e44253129884f1db54fc2f4406ea2462924ee3c115e5e86", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T9", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T9" }, "html": { "body": "

p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10), ‘Studies using election data’. The declared Gaussian test has the stated normal power, and reducing its standard error by the cluster-count factor increases power and gives the stipulated arithmetic corrections.

", "headline": "Power analysis, p. 9: ‘The power analyses presume that p-values will be constructed with the bias adjustment and robust standard errors as implemented by rdrobust’, with Tables 2 and 3, corrected for four clustered estimates whose saved power uses a standard error smaller than the fitted robust one (formal content: the normal power function, its strict monotonicity, the cluster-count factor, and the arithmetic of the six affected cells)", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10), ‘Studies using election data’" }, "id": "P033-T9", "text": { "body": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10), ‘Studies using election data’. The declared Gaussian test has the stated normal power, and reducing its standard error by the cluster-count factor increases power and gives the stipulated arithmetic corrections.", "headline": "Power analysis, p. 9: ‘The power analyses presume that p-values will be constructed with the bias adjustment and robust standard errors as implemented by rdrobust’, with Tables 2 and 3, corrected for four clustered estimates whose saved power uses a standard error smaller than the fitted robust one (formal content: the normal power function, its strict monotonicity, the cluster-count factor, and the arithmetic of the six affected cells)", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10), ‘Studies using election data’" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T9" }, { "declarations": [ { "claim_revision_id": "catalog:9442645fe8b9c0af2120bc7ae9fc606cbcdf2feb5a997c87df5c4920c52f0518", "conditional_on": [], "credit_reason": "This checks the two printed-description witnesses, not an error in the plotting code. The variance-versus-SD example shows different coordinate maps; it does not infer that the actual code divides by variance.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.funnel_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L174", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:9442645fe8b9c0af2120bc7ae9fc606cbcdf2feb5a997c87df5c4920c52f0518", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T10", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T10" }, "html": { "body": "

p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8). The printed funnel slope misclassifies the displayed point, and variance and standard-deviation scaling give different coordinates with the same t ratio.

", "headline": "p. 7: ‘The boundary for statistical significance at the 5% level is a line emanating from the origin with a slope of 1.96 […] where estimates to the right of this line attain statistical significance’, and ‘We use the sample variance of the outcome among control units for the standardization’, as printed; the witnesses are the validator's and are not in the paper", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8)" }, "id": "P033-T10", "text": { "body": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8). The printed funnel slope misclassifies the displayed point, and variance and standard-deviation scaling give different coordinates with the same t ratio.", "headline": "p. 7: ‘The boundary for statistical significance at the 5% level is a line emanating from the origin with a slope of 1.96 […] where estimates to the right of this line attain statistical significance’, and ‘We use the sample variance of the outcome among control units for the standardization’, as printed; the witnesses are the validator's and are not in the paper", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T10" }, { "declarations": [ { "claim_revision_id": "catalog:c757efd23ff3c7c41cf2d3aaec7515eb53e33d92dc4ad0ab9e3fe3a7982ca616", "conditional_on": [], "credit_reason": "The target explicitly requests the rational squared-endpoint enclosure. Its positive endpoints give the corresponding SD enclosure without identifying an irrational square root with a finite decimal.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.funnel_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L191", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:c757efd23ff3c7c41cf2d3aaec7515eb53e33d92dc4ad0ab9e3fe3a7982ca616", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T11", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T11" }, "html": { "body": "

p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8). Common scaling preserves the t ratio, significance has reciprocal boundary slope, and the binary control sample has the stated variance and standard-deviation enclosure.

", "headline": "p. 7, the two sentences describing the standardization and the significance boundary of Figure 4, corrected: the divisor is the sample standard deviation and the boundary has slope 1/z, about 1/1.96", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8)" }, "id": "P033-T11", "text": { "body": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8). Common scaling preserves the t ratio, significance has reciprocal boundary slope, and the binary control sample has the stated variance and standard-deviation enclosure.", "headline": "p. 7, the two sentences describing the standardization and the significance boundary of Figure 4, corrected: the divisor is the sample standard deviation and the boundary has slope 1/z, about 1/1.96", "location": "p. 7 (PDF p. 7), Results, right column, paragraph beginning ‘We can disaggregate the information in Figure 3’; Figure 4, p. 8 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T11" }, { "declarations": [ { "claim_revision_id": "catalog:bddd44db7601dfb0d990ae8b0574ec89e56a22e0372228d2674a58ccd47efb25", "conditional_on": [], "credit_reason": "This is a counterexample to all-level validity of the interval transform in the stipulated known-bound Gaussian model. The B=1/2 target asks for the unique root and over-rejection value rather than repeating the universal honesty clause; the general B honesty/5%-size result is separately credited in R211-13.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.pseudo_p_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/Wave1/Gap/GLP010P033P033B1.lean#L645", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:bddd44db7601dfb0d990ae8b0574ec89e56a22e0372228d2674a58ccd47efb25", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T12" }, "html": { "body": "

Supplement, Figure S9 (printed supplement p. 18) and Section S9 (printed supplement p. 16), text extraction only; main text p. 8 (PDF p. 8), right column, top paragraph. For bias bounds one and one half, exactly honest Gaussian intervals produce pseudo p-values that over-reject at level one fifth.

", "headline": "Supplement Figure S9 (vertical axis ‘RDHonest P−Value’, caption ‘Original p-values vs. p-values derived using RDHonest’), with the pseudo t-statistics of p. 8 of the main text: the normal-tail transform of a 95% bias-aware interval, read as a p-value; the witness is the validator's construction and is not in the paper", "location": "Supplement, Figure S9 (printed supplement p. 18) and Section S9 (printed supplement p. 16), text extraction only; main text p. 8 (PDF p. 8), right column, top paragraph" }, "id": "P033-T12", "text": { "body": "Supplement, Figure S9 (printed supplement p. 18) and Section S9 (printed supplement p. 16), text extraction only; main text p. 8 (PDF p. 8), right column, top paragraph. For bias bounds one and one half, exactly honest Gaussian intervals produce pseudo p-values that over-reject at level one fifth.", "headline": "Supplement Figure S9 (vertical axis ‘RDHonest P−Value’, caption ‘Original p-values vs. p-values derived using RDHonest’), with the pseudo t-statistics of p. 8 of the main text: the normal-tail transform of a 95% bias-aware interval, read as a p-value; the witness is the validator's construction and is not in the paper", "location": "Supplement, Figure S9 (printed supplement p. 18) and Section S9 (printed supplement p. 16), text extraction only; main text p. 8 (PDF p. 8), right column, top paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T12" }, { "declarations": [ { "claim_revision_id": "catalog:3eb787493006e4b373922cce96ccc0e843d905bb7dca41160a41c09b95a597ad", "conditional_on": [], "credit_reason": "Both strict and non-strict 5% decisions are distinguished correctly for a closed confidence interval. The known fixed bias bound and unit variance are part of the corrected mathematical target, not a claim about data-estimated bounds in the paper.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.Wave1.pseudo_p_correction_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/Wave1/Gap/GLP010P033P033B1.lean#L877", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:3eb787493006e4b373922cce96ccc0e843d905bb7dca41160a41c09b95a597ad", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T13" }, "html": { "body": "

Supplement, Figure S9 (printed supplement p. 18), text extraction only; main text p. 8 (PDF p. 8). The pseudo p-value preserves the interval decision at 5%, and the specified worst-case Gaussian tail is valid at every level.

", "headline": "Supplement Figure S9, corrected: the vertical quantity is a pseudo p-value, the normal-tail transform of the pseudo t-statistic of p. 8; it reproduces the 5% decision of the 95% interval and is not in general a valid p-value at other levels", "location": "Supplement, Figure S9 (printed supplement p. 18), text extraction only; main text p. 8 (PDF p. 8)" }, "id": "P033-T13", "text": { "body": "Supplement, Figure S9 (printed supplement p. 18), text extraction only; main text p. 8 (PDF p. 8). The pseudo p-value preserves the interval decision at 5%, and the specified worst-case Gaussian tail is valid at every level.", "headline": "Supplement Figure S9, corrected: the vertical quantity is a pseudo p-value, the normal-tail transform of the pseudo t-statistic of p. 8; it reproduces the 5% decision of the 95% interval and is not in general a valid p-value at other levels", "location": "Supplement, Figure S9 (printed supplement p. 18), text extraction only; main text p. 8 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T13" }, { "declarations": [ { "claim_revision_id": "catalog:750b388ddad16e1bfada8203fee26cda8f1661f794b6afbb9e84d2c7834436b9", "conditional_on": [], "credit_reason": "The endpoints and released-code provenance are supplied empirical premises; the theorem faithfully proves their algebraic consequence and does not claim to recover metadata from files.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.interval_rule_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L220", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:750b388ddad16e1bfada8203fee26cda8f1661f794b6afbb9e84d2c7834436b9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T14", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T14" }, "html": { "body": "

Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column, end of the paragraph on Figure 1. The two interval-imputation rules differ on the recorded inputs and put the second statistic in adjacent histogram bins.

", "headline": "Supplement Section S4: standard errors imputed from ‘the upper value of the reported … confidence interval’ and the point estimate (Ariga; Holbein; Klašnja and Titiunik), with the main-text sentence of p. 4, against the rule that produced the published statistics; the witnesses are rows of the deposited metadata and are not in the paper", "location": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column, end of the paragraph on Figure 1" }, "id": "P033-T14", "text": { "body": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column, end of the paragraph on Figure 1. The two interval-imputation rules differ on the recorded inputs and put the second statistic in adjacent histogram bins.", "headline": "Supplement Section S4: standard errors imputed from ‘the upper value of the reported … confidence interval’ and the point estimate (Ariga; Holbein; Klašnja and Titiunik), with the main-text sentence of p. 4, against the rule that produced the published statistics; the witnesses are rows of the deposited metadata and are not in the paper", "location": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column, end of the paragraph on Figure 1" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T14" }, { "declarations": [ { "claim_revision_id": "catalog:17e4ca0f8d140037aa9825d1f4812dc170a5250573a62e95759754a1a95238ba", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P033/P033_Audit.lean:274 specifies the corrected width-based imputation, actual Gaussian quantile choices, absolute reconstructed statistic and missing-p transform, then proves the midpoint comparisons, centered-interval recovery and recorded numerical formulas of T15. The initial equalities unfold the reconstruction definitions, which is appropriate for a claim specifying an imputation rule; the remaining conclusions are substantive algebraic statements, so the theorem is not vacuous. Its positive-quantile guard matches the target's q>0 assumption and holds for the stated 90% and 95% Gaussian quantiles, while all comparison and recovery clauses quantify arbitrary positive q. It asserts no sampling-validity conclusion for arbitrary reported intervals, and none is required by this corrected target.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P033.interval_rule_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P033/P033_Audit.lean#L270", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:17e4ca0f8d140037aa9825d1f4812dc170a5250573a62e95759754a1a95238ba", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T15" }, "html": { "body": "

Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column. Width-based imputation has the stated reconstruction formulas, recovers centered normal-interval standard errors, and agrees with the upper-endpoint rule exactly at midpoint estimates.

", "headline": "Supplement Section S4 (Ariga; Holbein; Klašnja and Titiunik) and the main-text sentence of p. 4, corrected: an interval-imputed standard error is the full width of the reported interval divided by twice the normal critical value", "location": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column" }, "id": "P033-T15", "text": { "body": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column. Width-based imputation has the stated reconstruction formulas, recovers centered normal-interval standard errors, and agrees with the upper-endpoint rule exactly at midpoint estimates.", "headline": "Supplement Section S4 (Ariga; Holbein; Klašnja and Titiunik) and the main-text sentence of p. 4, corrected: an interval-imputed standard error is the full width of the reported interval divided by twice the normal critical value", "location": "Supplement, Section S4 (printed supplement pp. 9–10), text extraction only; main text p. 4 (PDF p. 4), right column" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T15" }, { "declarations": [ { "claim_revision_id": "catalog:29fd601573e701cdb3e949742ae1ff8e52ec764f44cef59afcdf29afd9b6798c", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:29fd601573e701cdb3e949742ae1ff8e52ec764f44cef59afcdf29afd9b6798c", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T16", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T16" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "The diagnosis of original inference, as printed: Abstract (‘rendering standard errors artificially small’), Introduction (p. 2), Results on Panel 3(c) (p. 7) and Concluding remarks (‘More than half of the studies in our sample’, p. 10)", "location": "Abstract, p. 1 (PDF p. 1); p. 2 (PDF p. 2), left column; p. 7 (PDF p. 7), Results; p. 10 (PDF p. 10), Concluding remarks" }, "id": "P033-T16", "text": { "body": "No current theorem is supplied for this target.", "headline": "The diagnosis of original inference, as printed: Abstract (‘rendering standard errors artificially small’), Introduction (p. 2), Results on Panel 3(c) (p. 7) and Concluding remarks (‘More than half of the studies in our sample’, p. 10)", "location": "Abstract, p. 1 (PDF p. 1); p. 2 (PDF p. 2), left column; p. 7 (PDF p. 7), Results; p. 10 (PDF p. 10), Concluding remarks" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T16" }, { "declarations": [ { "claim_revision_id": "catalog:f5c752b973e677003b006cfe4d16714b8c250a3750c9311c026019997ce95988", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:f5c752b973e677003b006cfe4d16714b8c250a3750c9311c026019997ce95988", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T17", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T17" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "The clustered reanalysis with the default settings of rdrobust and the power entries that rest on it, as printed: Methodology (p. 7), Power analysis and Table 2 (p. 9), Table 3, first panel (p. 10)", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; p. 10 (PDF p. 10), Table 3" }, "id": "P033-T17", "text": { "body": "No current theorem is supplied for this target.", "headline": "The clustered reanalysis with the default settings of rdrobust and the power entries that rest on it, as printed: Methodology (p. 7), Power analysis and Table 2 (p. 9), Table 3, first panel (p. 10)", "location": "p. 7 (PDF p. 7), Methodology; p. 9 (PDF p. 9), Power analysis and Table 2; p. 10 (PDF p. 10), Table 3" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T17" }, { "declarations": [ { "claim_revision_id": "catalog:b6d87f003c9e8ad30a8bf4abdb48edcb8745d526826797b07a086aba3892da20", "conditional_on": [], "credit_reason": "Historical excluded association.", "credit_status": null, "fidelity_status_baseline": null, "lean_name": null, "manifest_use": "not_applicable", "named_axioms": [], "says": null, "source": null, "status": "not_applicable", "support_scope": "historical declaration", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:b6d87f003c9e8ad30a8bf4abdb48edcb8745d526826797b07a086aba3892da20", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P033-T18", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P033-T18" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "The Cohen's d standardization of the power analysis and the two cells it affects, as printed: p. 9 (definition; Table 2, row 0.1, column 95) and Table 3, second panel (row 0.1, column 95)", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10)" }, "id": "P033-T18", "text": { "body": "No current theorem is supplied for this target.", "headline": "The Cohen's d standardization of the power analysis and the two cells it affects, as printed: p. 9 (definition; Table 2, row 0.1, column 95) and Table 3, second panel (row 0.1, column 95)", "location": "p. 9 (PDF p. 9), Power analysis, left column, and Table 2; Table 3, p. 10 (PDF p. 10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P033/#P033-T18" } ] }, "missing_sources": [ "Original PDF of the official 23-page online supplement (sj-pdf-1-rap-10.1177_20531680231166457.pdf)" ], "reproduction": { "agreement": "to_rounding", "status": "partial", "targets": [ { "agreement": "to_rounding", "location": "p. 5 (PDF p. 5)", "method": "authors' code as shipped", "status": "reproduced", "target": "Table 1" }, { "agreement": "to_rounding", "location": "p. 5 (PDF p. 5); sample sizes p. 4", "method": "authors' code as shipped", "status": "reproduced", "target": "Figure 1" }, { "agreement": "to_rounding", "location": "p. 6 (PDF p. 6)", "method": "authors' code as shipped", "status": "reproduced", "target": "Figure 2" }, { "agreement": "to_rounding", "location": "p. 8 (PDF p. 8); sample sizes and interpretation p. 7", "method": "I reran the rdrobust analyses on deposited cleaned data, omitting unavailable rdpower and RDHonest calculations. The 39 plotted estimates from 25 studies match the deposited results within floating-point precision, and the regenerated panels match the printed figure.", "status": "reproduced", "target": "Figure 3" }, { "agreement": "to_rounding", "location": "p. 8 (PDF p. 8); methodology and sample sizes p. 7", "method": "I reran the rdrobust analyses on deposited cleaned data and compared the standardized coordinates with the deposited results and printed panels. Removing I() wrappers restores the coordinates with ggplot2 4.0.2; this leaves the data and arithmetic unchanged.", "status": "reproduced", "target": "Figure 4" }, { "agreement": "to_rounding", "location": "p. 9 (PDF p. 9)", "method": "I aggregated the deposited individual powers with the article weights; every printed cell agrees to rounding. I did not rerun rdpower. P033-E010, P033-N010 and P033-N011 give corrections and conditional alternatives to the underlying power calculations.", "status": "recalculated", "target": "Table 2" }, { "agreement": "to_rounding", "location": "p. 10 (PDF p. 10); election shares p. 9", "method": "I aggregated the deposited individual powers with the article weights; every printed cell agrees to rounding. I did not rerun rdpower. P033-E010, P033-N010 and P033-N011 give corrections and conditional alternatives to the underlying power calculations.", "status": "recalculated", "target": "Table 3" }, { "agreement": "to_rounding", "location": "p. 10 (PDF p. 10)", "method": "authors' code as shipped", "status": "reproduced", "target": "Figure 5" } ] }, "summary": "The paper asks how reliable published regression discontinuity (RD) findings in political science are. For the 45 RD studies published in three journals from 2009 to 2018, it tests whether $t$-statistics bunch just above $1.96$; it reanalyzes the 36 studies with available data by one automated procedure and computes their power. It reports bunching, larger standard errors on reanalysis, and low power. 14 entries are classified as errors (1 Major, 2 Moderate, 11 Minor), and 2 are clarity notes; 5 other allegations were not confirmed and 3 remain undetermined. The general warning about neglected smoothing bias is sound. The larger replacement standard errors do not establish which original procedures failed their coverage conditions or the asserted majority (\\fid{P033-N006}). For Coppock and Green, the count-scale standardization is reproducible but differs from individual-turnout Cohen's $d$ (\\fid{P033-N010}). The clustered variance cancellation is inherited from rdrobust's defaults and requires qualification of their use as an uncertainty benchmark (\\fid{P033-N011}); the broad low-power concern survives.", "what_was_checked": "I checked the following, which held up within the limits stated below.\n\\begin{itemize}\n\\item The sample counts: 45 studies, 80 primary $t$-statistics ($1.78$ per article), 36 studies and 64 analyses with replication data, and 39 estimates from 25 studies in the main reanalysis. Election data are used by $31/45\\approx68.9\\%$ of all studies and by $26/36\\approx72.2\\%$ of those with data.\n\\item Table~1: the below/above counts $(3,9)$, $(4,11)$ and $(5,14)$, the window radii $0.196$, $0.294$ and $0.392$, and the one-sided binomial tails $0.073$, $0.059$ and $0.032$ at probability $1/2$ (see \\fid{P033-N004} for their interpretation). Tables~S4a, S4b, S5a and~S5b also reproduce, and the counts in Tables~S4a and~S4b sum to those in Table~1 at every caliper.\n\\item Tables~2 and~3 as coded: every entry reproduces from the stored power values with the article weights. Corrections and their limits are given in \\fid{P033-E010}, \\fid{P033-N010} and \\fid{P033-N011}; the last two give conditional changes for each study separately.\n\\item The arithmetic and totals of Tables~S2 and~S3 (see \\fid{P033-E009} for the year of one study), the five rows of Table~S6, and Table~S1 and the list of nine studies without data, which agree with the metadata.\n\\item The rdrobust reanalysis: running the reanalysis code on the estimates with data gives the stored coefficients and the stored conventional and robust standard errors.\n\\item The shift to the left in Figure~3 holds across its full sample of 39 estimates, where the weighted share of $t$-statistics at or above $1.96$ is $0.77$, $0.63$ and $0.49$.\n\\item The stored RDHonest pseudo $t$-statistics equal the absolute estimate divided by the half-length over the standard-normal $97.5\\%$ quantile (approximately $1.96$), as described, and the pseudo standard errors exceed the original ones for 38 of 46 estimates (see \\fid{P033-E001} for Figure~S9).\n\\item The argument on p.~8 that, when power is close to the significance level, the share of false positives among significant findings is close to the share of true nulls.\n\\item The description on pp.~3--4 of the three approaches to bias in RD inference, apart from the wording noted in \\fid{P033-A001} and \\fid{P033-E002}.\n\\item The selections, weights and numerical axes of Figures~1 to~5 and~S1 to~S9 agree with their code and with the stored results, and regenerated Figures~1 to~5 match the printed panels (see \\fid{P033-N002} for what the axis limits leave out and \\fid{P033-E001} for the label in Figure~S9).\n\\item The 2025 erratum for this article only adds the link to the replication files, and the data files match their published checksums.\n\\end{itemize}\nI worked within the following limits. I did not inspect the typeset pages of the online supplement, apart from the passages shown in the exhibits. I read its extracted text and compared its tables and figure descriptions with the tables and figures in the replication files. The raw study data and intermediate cleaning files were unavailable, so I could not run the cleaning of the study data. I did not run the packages rdpower and RDHonest. I used the stored individual powers and RDHonest fits, compared the nonclustered powers with the closed-form formula, and recomputed powers for the four clustered estimates in \\fid{P033-E010}. With few exceptions I did not read the 45 reviewed articles or compare the extracted coefficients, standard errors and classifications with the originals. Four of the five numerical values attributed to other works were supported by the available copies, with limits on matching the cited versions and pages. The absence of further corrections does not establish that the rest of the paper is correct.\n\nThe paper's warning that neglected, nonnegligible smoothing bias can invalidate conventional inference is sound. The larger replacement standard errors are consistent with understated uncertainty, but do not identify which original procedures failed their coverage conditions or establish the asserted majority (\\fid{P033-N006}). The standardized count slope for Coppock and Green and its power of 1 are reproducible for that target; it differs from individual-turnout Cohen's $d$ (\\fid{P033-N010}). The within-cell clustered-variance cancellation in \\fid{P033-N011} comes from rdrobust's documented defaults, inherited by the paper's ordinary call, and requires qualification of their use as an uncertainty benchmark. The alternative residual calculations illustrate removal of that cancellation; they do not establish true power or coverage. The broad low-power concern and the general value of bias-aware methods survive, without establishing that every original study had valid or invalid inference.", "work": { "arxiv_id": "2109.14526", "audit_complete": true, "authors": [ "Drew Stommes", "P. M. Aronow", "Fredrik Sävje" ], "category": "Articles - Empirics", "citation": "Drew Stommes, P. M. Aronow, and Fredrik Sävje (2023). On the reliability of published findings using the regression discontinuity design in political science. Research and Politics 10(2):1–12.", "doi": "10.1177/20531680231166457", "edition": { "pages_in_audited_file": 12, "sha256": "66da20d4586339061fad920e4b764e07d4656395ea41e238d109c5f8d1cf8748", "source_url": "https://doi.org/10.1177/20531680231166457", "venue": "Research and Politics", "year": "2023" }, "id": "P033", "later_materials": null, "title": "On the reliability of published findings using the regression discontinuity design in political science", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P033/", "venue": "Research and Politics", "year": "2023" } }