Corrections to: On the reliability of published findings using the regression discontinuity design in political science

The problem

The three sentences state a diagnosis of the original studies: their methods of inference were inappropriate, their standard errors were too small, and this holds for more than half of the 45 studies in the sample. The paper supports the diagnosis with a comparison. It estimates 39 effects from 25 studies again with the default settings of rdrobust, and it compares the new standard errors and $t$-statistics with the published ones. The robust standard error of the reanalysis exceeds the original standard error for 27 of the 39 estimates, the absolute $t$-statistic is smaller for 28, and the number of estimates above $1.96$ falls from 30 to 18.

This comparison is between two estimators. To assess whether an original interval was appropriate for the RD argument, I need to assess its bias and sampling distribution, reported standard error and critical value under the maintained smoothness assumptions and any additional justified restrictions. Exact coverage at linear conditional means does not establish validity under those broader assumptions. The reanalysis changes the estimator and the bandwidth, adds a bias correction whose noise enters the standard error, and uses no covariates. These changes affect sampling variance. The finite example below has exact coverage at affine means and illustrates that variance comparison; it does not show validity across the smoothness class. Correct undersmoothing, which the paper explicitly accepts, can provide such validity without exact linearity, and a bias-corrected replacement may still have a larger standard error. The same comparison can also arise when the original interval almost never covers the effect. The comparison alone does not distinguish the two cases. This does not vindicate a method that lacks its claimed coverage guarantee under the maintained model; the paper need not establish actual curvature to criticize such a method. Its general warning about nonnegligible smoothing bias is sound. What still requires a connection to the original procedures is the empirical diagnosis and its stated prevalence.

The sentence on p. 10 needs a count. “More than half” of 45 studies means at least 23 studies with inappropriate inference. The paper does not classify the 45 studies by whether their inference is valid, and it does not say which count the sentence rests on. The relevant counts from the paper, the supplement and the replication files are listed in the second table below; none classifies inference validity. On pp. 3–4 the paper names three valid approaches: undersmoothing with conventional inference, bias correction with a standard error that accounts for it, and a worst-case adjustment for bias. If a study chose its bandwidth without an automated rule, or did not use rdrobust, it does not follow that its inference is inappropriate. Note that exactly 23 studies are coded with a non-automated bandwidth. The paper does not present this count as the basis of the sentence, and by the criteria of pp. 3–4 it does not count inappropriate inference.

The diagnosis is not shown to be false. The wording of p. 7 is evidential (“indicating”), rather than a claim that every increase in a replacement standard error proves original undercoverage. The theory and reanalysis support concern about inference that neglects smoothing bias. The descriptions of the original methods and the replacement comparisons do not establish how many original studies failed the relevant bias-control conditions, particularly the asserted majority of 45. The more limited empirical conclusion is that the reanalysis is consistent with understated uncertainty.

Evidence a derivation

The evidence is a derivation with exact examples, a comparison under bounded curvature and undersmoothing, a recalculation of the counts from the results saved in the replication files, and a check of the pages.

For the example, let the running variable $X$ be uniform on $(-5,5)$ with a sharp cutoff at $0$, and let $Y_i(0)=2+X_i+\varepsilon_i$ and $Y_i(1)=6+X_i+\varepsilon_i$, where the errors $\varepsilon_i$ are independent $N(0,1)$, independent of the scores, and their variance is known. The RD effect is $4$. I condition on eight observations with scores $\pm1,\pm2,\pm3,\pm4$ and compare three estimators. Each is the difference between a fitted intercept on the right and a fitted intercept on the left. The estimator $G$ fits a line to the four points on each side. The estimator $L$ is local linear with the triangular kernel and bandwidth $4$. The estimator $R$ subtracts from $L$ an estimate of its bias from a quadratic fit with the same kernel and bandwidth. With the points on one side ordered by their distance $1,2,3,4$ from the cutoff, the intercept weights of $G$, $L$ and $R$ are

Each vector of weights sums to $1$ and has first moment $0$, which implies that each estimator is unbiased for $4$. Since the errors are independent with variance $1$, I have

Each estimator is a linear combination of independent normal errors. I conclude that $(G-4)/\sqrt3$, $(L-4)/\sqrt5$ and $(R-4)/\sqrt{38}$ are standard normal. For each of the three, the interval that extends $z_{.975}$ standard errors on either side of the estimate covers the effect with probability exactly $0.95$ at these specified affine conditional means and conditional on the eight scores. The standard error rises from $1.73$ to $2.24$ to $6.16$, with pointwise coverage unchanged. For the outcomes with errors $(1/10,-1/10,1/10,-1/10)$ on each side, all three estimates equal $4$, and the squared $t$-statistics are $16/3$, $16/5$ and $8/19$. Since $16/3>(1.96)^2=2401/625>16/5>8/19$, the first estimate is significant at $1.96$ and the two replacements are not.

For the opposite case, I define $\psi(x)=(x^2-1)(x^2-4)(x^2-9)(x^2-16)/576$. This polynomial vanishes at the eight scores, and $\psi(0)=1$. Let the mean of $Y(1)$ be $6+x+20\psi(x)$ instead. The effect is now $24$, and the distribution of the eight outcomes is unchanged, so that the three estimates, their standard errors and the comparison between them are unchanged. The interval $G\pm1.96\sqrt3$ covers $24$ only if $|G-4|>16$. By Chebyshev’s inequality, this event has probability at most $3/256$. The comparison is the same in the two cases. Coverage is exactly $95\%$ at the first, affine pair of means and at most $1.2\%$ at the second pair. These pointwise calculations show different actual coverage, not a coverage guarantee over a common smoothness class. Even a small quadratic perturbation, $\mathbb{E}[Y(1)\mid X=x]=6+x+ax^2$ with $a>0$, biases $G$ by $-5a$. At $a=1/10$ its conditional coverage is about $0.9404$, although the curvature is only $1/5$. Exact coverage at the affine means therefore cannot rebut the paper’s smoothness-based criticism.

A further example concerns covariates. Suppose that observed covariates $Z_i\sim N(0,3)$, independent across observations and of the scores and errors, enter both potential outcomes with known coefficient $1$. Then the estimator $G$ applied to $Y-Z$ has variance $3$, and applied to $Y$ it has variance $12$. Both intervals have exact coverage at the stipulated covariate and affine data-generating process, and omitting the adjustment doubles the standard error. This does not establish validity for an original study under its maintained assumptions. The main call to rdrobust in the replication files passes no covariates, and the original estimate of Gerber and Hopkins (2011) is coded there as adjusted for covariates.

There is also a comparison with valid inference over a smoothness class. Take scores $x_{+,j}=j/n$ and $x_{-,j}=-j/n$, $j=1,\ldots,n$, independent $N(0,1)$ errors of known variance, and twice differentiable conditional means with $|m_d''(x)|\le M$ for a fixed finite $M$. Put $m=\lfloor n^{3/4}\rfloor$ and $h=(m+1)/n$. Let $A$ be the local linear difference at the cutoff, using the first $m$ observations on each side with triangular weights $1-j/(m+1)$. Let $B$ subtract its estimated quadratic Taylor bias using the same bandwidth and kernel. The weighted normal equations make $B$ the difference of the local quadratic intercepts.

For $u=j/(m+1)$, the limiting intercept weight functions are

Their squared integrals are $24/5$ and $72/7$. Both fits reproduce constants and lines and have bounded sums of absolute intercept weights, so Taylor’s theorem bounds each effect estimator’s bias by a constant times $Mh^2$. Their variances are asymptotic to $2(24/5)/m$ and $2(72/7)/m$. Thus the absolute bias divided by its standard deviation is $O(h^2\sqrt m)=O(n^{-1/8})$, uniformly over the stated mean class. The centered noise is exactly normal, so both conventional intervals have coverage tending uniformly to $0.95$. Yet $\operatorname{Var}(B)/\operatorname{Var}(A)\to15/7$ and the ratio of standard errors tends to $\sqrt{15/7}\approx1.464$; for a fixed nonzero effect, the ratio of absolute test statistics for $B$ and $A$ tends in probability to $\sqrt{7/15}<1$. This establishes the paper’s undersmoothing exception without exact linearity. It does not reproduce any sampled study or the automatic bandwidths of rdrobust.

The 10 studies of the second row are 7 with an IK bandwidth and 3 with a cross-validated one, and the 23 studies of the third row include 4 with a global polynomial specification. Weighting each study equally, the mean standardized standard error rises from $0.188$ to $0.265$ and the mean absolute $t$-statistic falls from $5.03$ to $3.91$. The count of 16 studies holds for the main estimate of each study and for the mean ratio within a study, with different sets of 16. With the conventional standard error at the same bandwidth, the counts are 21 of 39 estimates and 12 of 25 studies.

The examples do not represent any study in the sample or the data-driven defaults of rdrobust. I did not examine the original articles or their code. The pointwise affine calculations do not validate the original methods under the paper’s smoothness assumptions, and the undersmoothing derivation does not show that the sampled studies used suitable bandwidths. The replacement comparison does not identify which original procedures failed their claimed coverage conditions or establish the asserted majority.

What survives

The descriptive comparison of the saved results stands: the counts 27 of 39 and 28 of 39, the fall from 30 to 18 estimates above $1.96$, and the larger standardized standard errors on average. The reading that this pattern is consistent with understated uncertainty survives. The general warning about nonnegligible smoothing bias on pp. 3–4 remains sound, as do the three valid routes to inference described there. The bunching of $t$-statistics, the caliper tests and the power analysis do not use the disputed prevalence claim. What changes is the empirical diagnosis in the abstract: it is qualified as a concern supported by the theory and the comparison, and the undocumented share “more than half” is withdrawn. The counts above use the saved values, including the erroneous Broockman and Ryan standard error addressed in P033-N011.

The problem

The definition is right in general. For Coppock and Green, the code accurately computes an outcome-standardized aggregate fuzzy-IV coefficient per additional prior voter, a different comparison from Cohen’s $d$ for individual turnout. In the analysis file for Coppock and Green (2016), each of the 140,659 rows is an aggregate: the outcome and the treatment are counts of voters, with maxima 294 and 552. The reanalysis is a fuzzy RD of one count on the other. Its estimate, $0.066$ with robust standard error $0.0155$, is a ratio of two discontinuities in counts. This is an aggregate outcome-count slope per prior voter; under the scientific interpretation in the paper’s study metadata, the ratio identifies an effect on an individual’s probability of voting. The code divides this effect by the standard deviation of the counts, $24.58$ within the bandwidth for the power analysis and $41.66$ below the cutoff for the funnel plots. The code’s $d=0.1$ passes an alternative of $2.46$ to the fuzzy-RD slope calculation, and the saved power equals $1$ at all four effect sizes. That value is consistent with the code-defined target and saved standard error; the mismatch concerns its interpretation as individual Cohen’s $d$. The divisor also depends on the size of the aggregates: at fixed bandwidths, doubling every count leaves the effect and its standard error unchanged and doubles the divisor.

For the intended individual voting contrast, the outcome is binary and its population standard deviation is at most $1/2$. An effect size of $d=0.1$ is then an effect of at most $0.05$, and under the normal approximation of the paper, with the saved robust standard error, the power at $d=0.1$ is at most about $0.8973$. Under this approximation, the study cannot attain 95% power at $d=0.1$, and one cell in each of Tables 2 and 3 changes. Whether the study attains 60% or 80% power at $d=0.1$ depends on the standard deviation of individual turnout, which the analysis file does not contain. The code also takes the observations to the left of the cutoff as the control units. In this file the treatment is concentrated on the left: within 200 days of the cutoff its mean is $100.7$ on the left and $0.42$ on the right. The left side is not a group of untreated individuals.

Evidence an exact counterexample

The evidence is an exact counterexample to interpreting the code-defined standardization as individual Cohen’s $d$, a recalculation from the replication files, and a bound on power for that individual target.

The counterexample has two types of cell on each side of an eligibility cutoff. The two types are equally likely, and each cell has 100 individuals. On the eligible side, each cell has 50 voters in the earlier election, and the two types have 10 and 50 voters in the later one; five of the earlier voters in each cell are induced to vote later, so the later-voter counts without treatment would be 5 and 45. On the ineligible side, no one voted in the earlier election, and the two types have 10 and 40 voters in the later one. The fuzzy RD effect is $(30-25)/(50-0)=1/10$. Turnout without treatment is $1/4$ on each side, with variance $3/16$, and Cohen’s $d$ satisfies $d^2=(1/10)^2/(3/16)=4/75$. The sample variance of the counts on the eligible side is $(10-30)^2+(50-30)^2=800$, and the standardization of the replication files gives $d^2=(1/10)^2/800=1/80000$. Doubling the population and the counts of every cell leaves the effect at $1/10$ and Cohen’s $d^2$ at $4/75$. The variance of the counts becomes $3200$, and the second quantity becomes $1/320000$.

For the bound, a binary outcome with mean $p$ has variance $p-p^2\le1/4$. Let $s$ be the saved robust standard error (approximately $0.0155$) and $z=z_{.975}$. Under the normal approximation, the power at $d=0.1$ with standard deviation $\sigma$ is

which increases in $\mu\ge0$. Substituting $\sigma=1/2$, I obtain approximately $0.8973$. Solving for the standard deviations such that the power equals 60%, 80% and 95%, I find approximately $0.343$, $0.434$ and $0.559$. The last exceeds $1/2$. A sample standard deviation with divisor $N-1$ can exceed $1/2$ slightly. For a control group that contains all untreated later-election voters in the cells to the right of the cutoff within the bandwidth, at least 31,731 individuals, the bound is approximately $0.89727$.

A fit of rdrobust version 2.2 to the analysis file reproduces the saved coefficient and robust standard error to within $10^{-8}$, with bandwidth $105.66$ days; the original code used version 2.1.1. The local count standard deviation is $24.57954623$, so the code-defined target at $d=0.1$ is $2.45795462$. These reproduce the alternative actually computed, not individual Cohen’s $d$. At the same bandwidths, doubling every count leaves the coefficient and its standard error unchanged. The study is one of the ten studies that do not use election data and has weight $1$. Changing its indicator for 95% power at $d=0.1$ from $1$ to $0$ changes the two cells as follows.

The bound holds for the normal approximation with the saved standard error. It is not a statement about the exact power of the test in finite samples. I take the saved powers from the replication files; I have not independently executed rdpower.

What survives

The estimate itself, its standard error, its $t$-statistic and its $p$-value do not use the standardization, and the histograms of $t$-statistics and the caliper tests are unchanged. A common divisor cancels in the ratio of an estimate to its standard error, so that the study stays on the same side of the significance line in the funnel plots. What changes is its row of the power analysis and its position on the common scale of Figures 4, S4, S7 and S8. In panel (a) of Figure 4 it is plotted at about $(0.002,0.00005)$. The advantage of studies without election data at $d=0.1$ becomes smaller and keeps its sign. If this study also failed the 60% and 80% thresholds, the proportions would be $0.30$ and $0.25$, against $1/13\approx0.08$ for election studies. Replacing the code-defined power of $1$ with power for the individual Cohen-$d$ target reinforces the conclusion that the studies are underpowered. The correction of P033-N011 lowers the same cell of Table 2 for another study; each value here applies one correction alone.

The problem

The reanalysis calls rdrobust with the cluster variable of each study and the default variance estimator, which is built from nearest-neighbour residuals. The cancellation arises in the package’s documented default combination of nearest-neighbour residuals and clustering, inherited through the paper’s ordinary call; the paper did not independently write the cancelling variance formula. I call a set of observations with the same value of the running variable a cell. When a cell has $m\ge4$ observations, the neighbours of each observation are the other $m-1$ members of the cell. The residual of an observation is then $\sqrt{m/(m-1)}$ times the deviation of its outcome from the mean of the cell. For any values of the outcomes, these residuals sum to zero over the cell. The regression weights depend only on the running variable and are constant on the cell. A cell that lies inside one cluster then contributes zero to the score of that cluster. The uncertainty shared by the members of the cluster is left out of the variance. If every cell in the estimation and bias bandwidths has at least four observations and lies inside one cluster, the estimated variance is zero for the conventional and for the bias-corrected estimator.

The data of Broockman and Ryan (2016) nearly have this structure. The 29,413 survey respondents share 759 distinct values of the running variable, and the clusters are 380 districts. Inside the selected bandwidth there are 1,516 respondents in 36 cells, and 1,470 of them are in cells contained in one district. The only cell with a nonzero contribution is one of 46 respondents shared by two districts. The saved standard errors, $0.00211$ for the conventional estimate and $0.00222$ for the robust one, come from this cell alone. The reanalysis $t$-statistic of $23.0$ and the saved power of $1$ at every effect size follow from them. With regression plug-in residuals in place of the nearest-neighbour residuals and the same clusters, the robust standard error is $0.0790$ at the saved bandwidths, with $t=0.65$, and $0.0330$ when the bandwidths are selected again, with $t=1.83$.

The paper uses the default procedure as its reanalysis benchmark. Its statement that the cited methods represent progress is a general methodological judgment and is not refuted here. The application of this default as an uncertainty benchmark needs qualification for clustered designs in which observations in a cluster share values of the running variable. The paper should have checked whether the documented default was suitable for those designs. A larger number of clusters does not remove the problem. I established the cancellation for one study of 36. Two estimates of Schickler et al., with cells of two or three observations, are sensitive to the residuals to a lesser degree. Three other clustered studies, Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al. (2016), were not checked, since I did not have their data files.

Evidence an exact counterexample

The evidence is a derivation, an exact counterexample, and a recalculation on the data of the replication files.

Let $C$ be a cell of $m\ge4$ observations with outcomes $y_i$ and sum $S$. The nearest-neighbour residual of observation $i$ is

which implies that $\sum_{i\in C}r_i=0$. The weight of an observation in the local polynomial fit depends only on its value of the running variable. The contribution of $C$ to the score of a cluster that contains it is this weight times $\sum_{i\in C}r_i$, which is zero. The same holds for the bias-corrected estimator.

The counterexample has ten independent clusters at $x=-5,\dots,-1,1,\dots,5$, with five observations each and cutoff $0$. The estimator is local linear with a quadratic fit for the bias, the triangular kernel and bandwidths $h=b=6$. Let $U_1,\dots,U_{10}$ be independent and uniform on $\{-1,1\}$, and let every observation of cluster $g$ have the outcome $U_g$ under both treatments. The RD effect is $0$. The conventional estimator equals $\sum_g a_gU_g$ with

and its variance is $\sum_g a_g^2=62/25$. A direct computation gives the variance $10$ for the bias-corrected estimator. Every nearest-neighbour residual is $0$, and both estimated variances are $0$ in all $1024$ outcomes, while the two estimates are nonzero in $912$ and $968$ of them. If independent noise, uniform on $\{-1,1\}$, is added to every observation, then the covariance matrix of each cluster is positive definite. The true variances become $372/125$ and $12$, and the estimated variances stay $0$.

More clusters do not help. For $n\ge3$, I place $n(n+1)$ clusters of five observations on each side at $x=\pm j/(n(n+1))$ and set $h=b=1/n$, so that $n$ clusters on each side have positive weight. With outcome $U_g$ for every observation in cluster $g$, without the added observation noise, the variance of the conventional estimator is

and the default estimate of it is $0$ for all $n$.

On the data of Broockman and Ryan, I find the same cancellation with exact rational arithmetic. In the union of the two bandwidth windows there are 43 cells on each side, each with at least four respondents, and all but one lie inside one district. The exception is the cell at $x=0.0013536513$ with 46 respondents from two districts, whose sums of deviations from the mean of the cell are $+19/46$ and $-19/46$. The sandwich formula applied to this cell alone reproduces both saved standard errors. A fit of rdrobust with the default variance reproduces the saved coefficients and standard errors of the three estimates in the figure to within $4.8\times10^{-16}$.

For the tables, I use the power under the normal approximation, $\Phi(d\,\sigma_0/s-z_{.975})+\Phi(-d\,\sigma_0/s-z_{.975})$, where $s$ is the robust standard error with plug-in residuals and $\sigma_0$ the standard deviation of the outcome among control units within the bandwidth. At $d=0.1$ the power of the Broockman and Ryan estimate is $0.089$ at the saved bandwidths and $0.294$ with bandwidths selected again, below all three thresholds in both cases. The study has weight $1$ of 36, and of 26 among election studies.

At the saved bandwidths, six further cells fall: the 60% and 80% cells at $d=0.2$ and the 95% cell at $d=0.5$, in Table 2 and in the election panel of Table 3. For Schickler et al., the robust standard error of estimate (b) at the saved bandwidths moves from $0.092$ to $0.122$ and its $t$-statistic from $2.00$ to $1.51$. The robust standard error of estimate (c) moves from $0.129$ to $0.153$.

Several limits apply. The true sampling standard deviations are unknown, and the ratio of the two standard errors compares one estimator with another, not an estimator with the truth. Plug-in residuals remove the cancellation. I do not claim that the resulting intervals have correct coverage or that this is the only repair. Computing the conventional standard error by weighted least squares with errors clustered by district, I obtain $0.0630$ at the saved bandwidth. The rdpower package was not available, and the powers above are the normal approximation. The paper used version 2.1.1 of rdrobust and the recalculation used version 2.2, which reproduces the saved values.

What survives

The statistics reported by the original studies, the selection of studies, the bunching of the original $t$-statistics and the caliper tests on those original statistics do not use the reanalysis variance. Reanalyses without this cancellation are unaffected by this finding, and so is the rdhonest reanalysis, which is restricted to studies without clustering. The Broockman and Ryan replacements lower the apparent precision and power of that reanalysis, reinforcing the conclusions on low power and on larger reanalysis standard errors. Survival of these broad conclusions does not validate the affected default as an uncertainty benchmark. What changes is the Broockman and Ryan reanalysis: its standard errors, its bandwidth, its $t$-statistics, its place in panels (b) and (c) of Figures 3 and 4, and its power. With it change the first rows of Table 2 and of the election panel of Table 3, the quoted 17%, and the summaries of the supplement that use the same row. P033-E010 concerns a different defect in the same tables and changes other cells. Under the stated normal approximation, P033-N010 lowers the same 95% cell of Table 2 for another study, and each value here applies one correction alone.

The problem

The quantity on the vertical axis is not a $p$-value returned by the RDHonest procedure. Let $\widehat\theta$ be the RDHonest estimate, let $h$ be the half-length of its 95% bias-aware confidence interval, and write $z=\Phi^{-1}(0.975)$. The replication code computes the pseudo standard error $h/z$ and the pseudo $t$-statistic $z|\widehat\theta|/h$, and it plots

The main text (p. 8) and Section S9 call the $t$-statistics “pseudo”, but the title, the axis label and the caption of Figure S9 call $p_*$ a $p$-value without qualification. A $p$-value $P$ for a null hypothesis must satisfy $\Pr(P\le\alpha)\le\alpha$ under the null for every level $\alpha$. The rule $p_*<0.05$ rejects exactly when the interval excludes zero. This implies that the rule has level at most 5% when the interval has 95% coverage. At other levels $p_*$ need not satisfy the requirement, as the example below shows.

Evidence an exact counterexample

This correction rests on a derivation, an exact counterexample and a comparison with the replication files. I checked the Figure S9 labels in the released figure and the caption in the supplement’s extracted text; I did not have the typeset supplement page. Since $\Phi$ is strictly increasing, I have for every $h>0$ that

For the counterexample, let $T\sim N(\theta+b,1)$, where the bias $b$ satisfies $|b|\le B$ for a known bound $B=1$. Let $H$ be the unique positive number such that

The interval $[T-H,T+H]$ covers $\theta$ with probability at least $19/20$ for every $\theta$ and every $|b|\le B$, with equality at $b=\pm B$. Applied to this interval, the transform is $p_*(T)=2\Phi(-z|T|/H)$. Fix $\theta=0$, $b=1$ and $\alpha=1/5$, and set $q=\Phi^{-1}(9/10)$ and $C=Hq/z$. Then $p_*\le 1/5$ exactly when $|T|\ge C$, and since $T\sim N(1,1)$, I have

In exact rational arithmetic, I find that $2.64614554821530<H<2.64614554821540$ and that

The transform therefore rejects a true null about 23.6% of the time at the nominal 20% level, although the worst-case coverage of the interval is exactly 95%. With $B=b=1/2$ the same probability exceeds $0.20414905632765$. At $\alpha=1/20$ the rejection probability is exactly $1/20$.

The 46 values plotted in Figure S9 agree with this transform of the stored pseudo $t$-statistics to within $1.3\times 10^{-15}$. For each of them, $p_*<0.05$ agrees with the 95% interval excluding zero, which 20 of the 46 intervals do. I have not established the rejection rates of the transform for the 46 estimates themselves. When there is no bias, so that $B=0$ and $H=z$, the transform is the ordinary two-sided normal $p$-value at every level.

What survives

No plotted number, interval or 5% decision changes. The pseudo $t$-statistics and pseudo standard errors of Figures S5 to S8 and Table S6 are as the paper describes them. The rdrobust results, the caliper tests and the power tables use other quantities, and Figure S9 is the only place where these values are used. The reading of the full vertical scale of Figure S9, and of its diagonal, as a comparison of $p$-values does change.

The problem

The paper says that the power calculation uses the robust standard error of rdrobust. For four sharp clustered estimates, from Gerber and Hopkins (2011), Hall and Thompson (2018), Lopes da Fonseca (2017) and Sances (2017), the saved power does not match this description. Let $s$ be the robust standard error of the rdrobust fit. Let $G$ be the number of distinct clusters, and let $G_L$ and $G_R$ be the numbers of clusters with observations to the left and to the right of the cutoff. All counts are taken after incomplete observations are removed. Each saved power equals the power under the normal approximation with the standard error

in place of $s$. If there exists a cluster with observations on both sides of the cutoff, I have $G<G_L+G_R$. The standard error used for power is then smaller than $s$, and the power is too large. Two of the four estimates cross a threshold of the tables. Each has weight $1$, and each crossing lowers a cell of Table 2 by $1/36$ and a cell of the election panel of Table 3 by $1/26$.

Evidence a recalculation from the source

The evidence is a recalculation from the replication files. Under the normal approximation of p. 9, the test statistic has distribution $N(\tau/s,1)$ when the effect is $\tau$. With $a=|\tau|/s$ and $z=z_{.975}$, the power of the two-sided 5% test is

where $\sigma_0$ is the standard deviation of the outcome among control units within the bandwidth. For $a>0$, I have $f'(a)=\varphi(a-z)-\varphi(a+z)>0$, which implies that the power is strictly larger for a smaller standard error. I had the data of 45 of the 64 estimates. Fitting rdrobust again for these 45 reproduces the saved coefficients and standard errors to a relative error of $4.3\times10^{-15}$. The 180 saved powers of these estimates equal $f(|\tau|/(s\sqrt{\rho}))$ to within $7\times10^{-16}$, where $\rho=1$ without clusters and $\rho=G/(G_L+G_R)$ with clusters.

The powers of Gerber and Hopkins and of Hall and Thompson are also too large, by at most $0.04$, and cross no threshold. The largest reduction among the four estimates is $0.185$. As exact fractions, the Table 2 cells change from $31/48$ to $89/144$, from $113/144$ to $109/144$ and from $31/48$ to $89/144$, and the Table 3 cells from $61/104$ to $57/104$, from $81/104$ to $77/104$ and from $61/104$ to $57/104$. All 36 printed cells are reproduced from the saved powers before the change.

Two limits apply. The source of the rdpower package was not available, and I do not say which of its lines produce the factor. The data files of seven further clustered estimates, from Bohlken (2018), Gulzar and Pasquale (2017) and Larreguy et al. (2016), belong to the replication files, but I did not have them. Their powers were not recomputed, and further cells may change. Note that the estimate of Broockman and Ryan (2016) also has clusters on both sides ($G=380$, $G_L=379$, $G_R=380$), but its powers round to $1$ with either standard error. Its standard error is the subject of P033-N011.

What survives

Coefficients, standard errors, $p$-values, $t$-statistics, the caliper tests and the funnel plots do not use the saved powers. The other 30 cells are unchanged by this correction, including the 80% column, so that the percentages quoted in the text (22%, 56% and 70%) stand. The correction lowers power and reinforces the conclusion that the studies are underpowered. It restores the standard error that the paper describes. It does not show that this standard error is right for every clustered design.

The problem

The paragraph misstates two features of Figure 4. The figure and the code that draws it are right. First, the estimates and their standard errors are divided by the sample standard deviation of the outcome among observations below the cutoff, not by the sample variance. Second, the figure has the standardized estimate on the horizontal axis and the standardized standard error on the vertical axis. On these axes the line $|t|=1.96$ has slope $1/1.96$, the reciprocal of the printed value. A line of slope $1.96$, with the rule that estimates to its right are significant, would classify estimates with $|t|<1.96$ as significant.

Evidence a historical Lean attribution (Generated status: clause support not certified in this release)

The evidence is a derivation, an exact counterexample to the printed slope, and a check of the code and of one study in the replication files. Let $s_0>0$ be the sample standard deviation of the outcome among control units, $\widehat\tau$ an estimate and $\widehat{\mathrm{SE}}>0$ its standard error. The figure plots

so that $x/y=|\widehat\tau|/\widehat{\mathrm{SE}}=|t|$. Let $z=z_{.975}$. Since $y>0$ and $z>0$, I have $|t|\ge z$ if and only if $y\le x/z$. The boundary is the line $y=x/z$ through the origin, with slope $1/z\approx0.5102$, and the significant estimates lie to its right.

For the counterexample, let $z=1.96=49/25$. The point $(x,y)=(1/2,1/2)$ has $|t|=1<1.96$. At height $y=1/2$ the line of slope $49/25$ passes through $x=25/98<1/2$. The point lies to the right of that line and is not significant.

The plotting script of the replication files draws the line with slope 1/qnorm(0.975) and divides both coordinates by y0_sd. The analysis script computes this divisor with sd() over the observations below the cutoff. For Boas and Hidalgo (2011), these are 568 observations with 82 ones and 486 zeros. Their sample variance is $123/994\approx0.124$, and the saved divisor agrees with $\sqrt{123/994}\approx0.3518$. I checked this one study only.

What survives

Figure 4, its red line, the estimates, the standard errors and every classification as significant are unchanged. The power analysis uses its own standard deviation, that of control units within the default bandwidth (p. 9), which this correction does not touch. Only the two terms in the description change.

The problem

Two cross-references point to the wrong place. Section S8 of the supplement is “All studies with replication data”. The results of the rdhonest reanalysis are in Section S9, “Re-analysis using rdhonest package”. The reference to Section S8 on p. 7, for the reanalysis of all 36 studies, is right. The supplement has no Table A2 or A3. The note stands below Tables S2 and S3 and explains their totals, 45 and 46.

Evidence a check of the printed page

The evidence is a check of the page for the main text. For the supplement I had the text of the official file and the page image shown above, not the PDF itself. The headings of Sections S8 and S9 and the titles and totals of Tables S2 and S3 are as stated. The table files of the replication files carry the same two tables.

What survives

Correcting these labels changes no number or result.

The problem

The two tables count Schickler, Pearson and Feinstein in 2009. Table S1 and the reference list of the supplement cite the study as 2010, in The Journal of Politics 72(3), 672–689. According to the publisher, the article appeared online on 12 July 2010 and in the print issue of July 2010, and no date in 2009 is listed. The study therefore belongs in 2010 whether an article is dated by its first online publication or by its issue. In the replication code the year is read from the internal label of the article, schickler_etal_2009. The study is in the JOP row of Table S2, the Non-automated row of Table S3 and both All rows. In each of these four rows the 2009 entry is one too high and the 2010 entry is one too low.

A second study is counted in a year other than its cited year. Cavaillé and Marshall is cited as 2019, in American Political Science Review 113(1), and is counted among the seven APSR articles of 2018. It appeared online on 15 November 2018 and in the issue of February 2019. Its count in 2018, and its place in the period 2009–2018, are correct if articles are dated by first online publication. The paper does not state this rule.

Evidence a recalculation from the source

This correction rests on a recalculation from the replication files and on the publisher’s dates. When the table code is run unchanged, it reproduces Tables S2 and S3. When only the year of this study is changed to 2010, the eight highlighted entries change and no others, and Tables 2, 3 and S6, which the same script produces, do not change. The entries of Tables S2 and S3 are taken from the extracted text of the supplement and from the tables in the replication files, which agree; I did not have the typeset supplement pages.

What survives

All row totals, the sample of 45 studies, and every estimate and weight are unchanged, since no analysis uses the year of publication. Only the eight annual counts change.

The problem

Each of the five phrases has a wrong, missing or repeated word. The meaning is clear in every case.

Evidence a check of the printed page

I checked the pages for the two phrases of the main text. The sentence after the first one prints “The worst-case is taken with respect to a stipulated class of functions”, and p. 8 prints “worst-case bias adjustment”. I compared the three supplement phrases in its extracted text with the marked images of those passages accompanying this entry; they agree. I did not have the typeset supplement pages for comparison.

What survives

Everything else is unaffected, since no statement of the paper depends on these words.

The problem

The rows of the first column are the four values of Cohen’s $d$ defined on p. 9, in units of the standard deviation of the outcome. They are not percentages. The percent sign belongs only to the columns, which are the power thresholds 60, 80 and 95 under the heading “Power (%)”. No entry of the tables needs rescaling.

Evidence a check of the printed page

The evidence is a check of the pages and of the replication files. The table files there carry the heading “Effect size”. The analysis script multiplies each value of $d$ by the standard deviation of the outcome and does not convert it to a percentage.

What survives

All entries of the tables, the power calculations and the comparisons drawn from them are unchanged.

The problem

The code for the histograms sets the horizontal axis to the interval from $0$ to $2\Phi^{-1}(0.975)$, which is approximately $[0,3.92]$. A $t$-statistic above this limit enters no bin, and the density of each panel is then normalized to the weight that remains in the interval. Figure 1(a) shows 37 of the 45 main results, and each of its bars is higher by the factor $45/37$ than it would be in a density of all 45. Let $W$ be the total weight of the sample named in the text, let $V$ be the weight in the interval, and let $B_j$ be the weight in bin $j$, where each bin has width $w=\Phi^{-1}(0.975)/18$. The printed height of bin $j$ is $B_j/(wV)$, and the height in a density of the full sample is $B_j/(wW)$. The printed heights are those of the distribution conditional on $t\le 2\Phi^{-1}(0.975)$.

The share of weight left out differs across panels. In Figure 3 it is 21%, 26% and 29% in panels (a), (b) and (c). The three panels that the text compares are conditioned on different sets. The funnel plots of Figure 4 have axis limits $0\le x\le 2$ and $0\le y\le 0.8$, and panels (a), (b) and (c) leave out 3, 3 and 4 of the 39 estimates. The main text, the captions and the text of the supplement state neither the omissions nor the normalization. The range of each axis is visible in the figures, and the counts 45, 80 and 39 given in the text are the correct sizes of the samples that enter the code.

Evidence a recalculation from the source

This correction rests on a recalculation from the replication files and on a measurement of the published figure. The table lists what each panel leaves out, computed from the stored $t$-statistics and the weights of the paper.

The bars of Figure 1(a) in the published PDF represent 37 results in total. A bar that holds one result has measured height $0.2482$. This agrees with $1/(37w)=0.2482$ and not with $1/(45w)=0.2041$. The figures in the replication files contain 36, 36 and 35 points for the three panels of Figure 4.

What survives

Within a panel, the order of the bars, the position of the spike near $1.96$ and the ratios between bars are unchanged, since every bar of the panel is rescaled by the same constant. Table 1 and Tables S4 and S5 use their designated main-estimate samples without these plotting limits, and their windows end below the axis limit. Tables 2 and 3 use all 64 analyses. In the full data the weighted share of $t$-statistics at or above $1.96$ is $0.77$, $0.63$ and $0.49$ for the samples of Figure 3(a), (b) and (c), so this decline in the share of significant results holds without the axis limit. The absolute scale of the densities and comparisons of bar heights between panels do change.

The problem

The description of how missing $t$-statistics were reconstructed differs from the replication code in two respects. First, the code fills a missing standard error from a reported $t$-statistic, from a 90% or 95% confidence interval, or from the replication data, and then computes every missing $t$-statistic as the absolute estimate divided by the standard error. No $t$-statistic is recovered from a reported $p$-value. Second, for a reported interval $[L,U]$ and point estimate $b$, the code computes the standard error as $(U-L)/(2q)$, where $q$ is $\Phi^{-1}(0.95)$ for a 90% interval and $\Phi^{-1}(0.975)$ for a 95% interval. Section S4 instead describes the rule $(U-b)/q$ for Ariga, for Holbein, and for Klašnja and Titiunik. The two rules agree only when the interval is centered on the estimate. Of the six estimates to which Section S4 applies this rule, four have a recorded interval that is not exactly centered. For three of them the two standard errors differ by less than 1%, and for Klašnja and Titiunik they differ by the factor $8/7$.

Evidence an exact counterexample

This correction rests on a derivation, an exact example and a recalculation from the replication files. I checked the Supplement Section S4 quotations against its extracted text; I did not have the typeset supplement pages. Set $s_W=(U-L)/(2q)$ and $s_U=(U-b)/q$. Then

which implies that $s_W=s_U$ if and only if $b=(U+L)/2$. For Klašnja and Titiunik the recorded values are $b=-0.15$ and $[L,U]=[-0.213,-0.101]$. Then $U-L=0.112$ and $U-b=0.049$, and I obtain $s_W/s_U=8/7$ for every $q>0$. At the printed critical value $q=1.96$, I have $s_W=1/35$ and $s_U=1/40$, and the absolute $t$-statistics are $21/4=5.25$ and $6$. The replication code, which uses $q=\Phi^{-1}(0.975)$, gives $5.2499$.

Of the 81 rows of the raw metadata, 65 report a standard error, 1 reports a $t$-statistic, 2 use a 90% interval, 10 use a 95% interval and 3 use replication data. After every reported $p$-value is erased from the raw metadata, the cleaning code returns the same 81 standard errors and $t$-statistics.

What survives

Correcting the description changes no numerical result, and the cleaning code reproduces the cleaned data exactly. The reanalysis, its caliper tests and the power calculations do not use the imputed standard errors. If the rule of Section S4 were applied instead, with its printed critical values, Tables 1 and S4 would keep their counts and $p$-values. One estimate of Holbein (2016) would move by one histogram bin in Figures 1(b), 2(b) and 5(b), and six entries of Table S6 would change.