Corrections to: Unbiased Estimation of the Average Treatment Effect in Cluster-Randomized Experiments

The problem

Section 6.2 states that the true effect in the scenario with heterogeneous effects is 0.7 percentage points, and Section 6.5 says that the dotted lines of Figure 4 mark it. In the published figure, the dotted lines stand at 1.00 on the axis in the four regression panels, and each printed bias equals the position of the solid line minus the position of the dotted line, to the printed decimal. The three design-based panels show no separate dotted line. The text and the figure cannot both be right, and rounding does not reconcile them: a value printed as 0.007 lies between 0.65 and 0.75 percentage points, and every dotted line lies between 0.99 and 1.01. If the true effect is 0.7, as the text states twice, then the biases of the four regressions are about 0.3 points higher than printed: about $+0.05$ for the difference-in-means, $+1.20$ and $+1.22$ for the two fixed-effects regressions, and $+2.32$ for the random-effects regression. The RMSE labels of the fixed-effects and random-effects panels ($1.8$, $1.8$ and $2.4$) are then too small. From the published means and standard deviations, I obtain about $1.9$, $1.9$ and a value between $2.6$ and $2.7$. The sentence that the difference-in-means is “only slightly biased downward” rests on the printed $-0.2$. Measured from 0.7, the plotted mean of that estimator is higher by about 0.05, with a Monte Carlo standard error of about 0.03, so the figure does not establish downward bias.

The label “Bias = 0.0” of the Horvitz–Thompson panel also needs to be recomputed. The mean drawn in that panel is 0.82, which exceeds every target between 0.65 and 0.75 by more than 0.05. If the target is exactly 0.7, the same holds for the Des Raj panel with $n_j$, whose mean is 0.77. These are disagreements between labels and plotted means in a simulation of 5,000 draws, and they do not imply that either design-based estimator is biased. The cause of the mismatch is not established, since the original simulation code and draws are not available.

Evidence a recalculation from the source

The evidence is a recalculation from the published PDF, in which Figure 4 is stored as vector artwork. The coordinates of the axis ticks, the solid lines and the dotted lines were read from the page and converted to the scale of the axis in exact rational arithmetic. The remaining ticks fit this calibration to within 0.0015 points, and the incompatibility with the stated effect persists with a tolerance of 0.005. A second, independent reading of the coordinates agrees with the first to within 0.002 points.

The solid line minus the dotted line is $-0.248$, $0.903$, $0.916$ and $2.025$ in the four panels, and these values round to the printed biases. Suppose that a printed bias $b$ is rounded to one decimal and that the solid line is at $m$. Then a target compatible with the label lies in $[m-b-0.05,\,m-b+0.05]$. For the four panels these intervals are $[0.903,1.003]$, $[0.954,1.054]$, $[0.965,1.065]$ and $[0.974,1.074]$, and each of them lies above $0.75$. For the RMSE, I use the identity

applied to the empirical distribution of the 5,000 estimates, with $\theta=0.7$, the plotted mean and a standard deviation within $0.05$ of the printed one. If the standard deviation uses divisor 4,999, its square is multiplied by $4{,}999/5{,}000$ to give the empirical variance. The last column of the table shows the resulting approximate range. The printed $2.3$ of the difference-in-means panel lies in its range, and the printed $1.8$, $1.8$ and $2.4$ lie outside theirs. The last column gives guides rather than exact replacement labels: the plotted positions and the printed standard deviations are rounded, and the original draws are not available.

What survives

The inconsistency concerns the reporting of Figure 4 and the downward-bias sentence of Section 6.5. With the published means and standard deviations and a target near 0.7, the biases of the fixed-effects and random-effects regressions remain positive, the random-effects regression still has the smallest standard deviation, and the Horvitz–Thompson estimator remains imprecise. The grouping by RMSE is unchanged: the smallest values belong to the Des Raj estimators, and then come the fixed-effects regressions, the difference-in-means, the random-effects regression and the Horvitz–Thompson estimator. On these numbers, the statement that the Des Raj estimators are the most precise in terms of RMSE stands. The arguments on bias and consistency in Section 3 and the estimators and unbiasedness results of Sections 4 and 5 do not use Figure 4. Figures 2 and 3 come from the separate calculation under the sharp null. Figure 5 concerns standard errors and does not depend on the placement of Figure 4’s reference lines. What changes is the size of the regression biases, the RMSE labels of three panels, and the statement that the difference-in-means shows a downward bias in this example; the empirical bias labels of the design-based panels also require checking. The theoretical result that the difference-in-means can be biased is not affected.

The problem

The paragraph on p. 40 measures efficiency by mean squared error. The marked sentence says that an estimator whose bias does not vanish must, in a large enough sample, have a larger mean squared error than the estimator with which it is compared, and the sentence on p. 42 draws the same conclusion from inconsistency. One sufficient condition is that the mean squared error of that comparator tend to zero, which neither sentence assumes. If asymptotic efficiency is defined so as to require consistency, the phrase on p. 42 holds by definition, but that reading gives no ranking by mean squared error and does not support the sentence on p. 40. The asymptotic regimes of the paper include the one in which the number of clusters is fixed and the clusters grow (p. 40 and Section 3.3, p. 47). In that regime the variance of an unbiased estimator need not vanish, and a biased estimator can keep the smaller mean squared error at every sample size.

Evidence an exact counterexample

The evidence is an exact counterexample, computed by enumerating every assignment in rational arithmetic. Let $M=3$, $m_t=1$ and $m_c=2$, and let the clusters have sizes $1$, $2$ and $4$. The control outcomes of the individuals are $(0)$, $(1,0)$ and $(1,0,0,0)$ in the three clusters, and every treatment outcome is $0$. Then $N=7$ and $\Delta=-2/7$. Each cluster is the treated one with probability $1/3$. Enumerating the three assignments, I obtain the following values of the estimators.

For a fixed $k$ the Des Raj estimator of Section 5.1 is unbiased, and I find that its mean squared error is $3/686+(9/14)(k-2/21)^2$. This implies that no fixed $k$ does better than $3/686$. The choice $k=0$ is the Horvitz–Thompson estimator. Now suppose that each individual is copied $h$ times within its cluster, as in the construction on p. 47. The cluster sizes and totals are multiplied by $h$, and $N=7h$. Every entry of the table is then the same for every $h$. I conclude that the difference in means has bias $-1/315$ at every sample size. It is inconsistent, since its error is at least $1/21$ in absolute value under every assignment. Its mean squared error is nevertheless smaller than that of the Horvitz–Thompson estimator and of the Des Raj estimator with any fixed $k$.

The corrected statement holds because mean squared error is the sum of the variance and the squared bias. Suppose that the bias $b_h$ of $T_h$ tends to $b\neq0$. Then $\operatorname{MSE}(T_h)\ge b_h^2>b^2/4$ for all large $h$. If also $\operatorname{MSE}(U_h)\to0$, then $\operatorname{MSE}(U_h)<b^2/4$ for all large $h$.

What survives

I found no use of the guarantee in the derivations checked. The bias formula and the replication arguments of Section 3 are unaffected, and so are the results on the Horvitz–Thompson estimator in Section 4. The same holds for the results of Section 5 on the Des Raj estimator: that it is unbiased, and the explicit comparison of its variance with that of the Horvitz–Thompson estimator. The rankings by root mean squared error in Sections 6.4 and 6.5 are computed for the application and do not rest on these sentences. The motivation survives in the weaker form of the next sentence of the paper: bias cannot always be ignored when efficiency is the concern. The counterexample does not show that biased estimators are better in general.

The problem

The paper defines $n_j$ as the number of individuals in cluster $j$ (p. 44), and Section 6.2 does not rescale it. For an individual in cluster $j$, the difference between the linear predictors under treatment and under control is $\tau+\phi n_j=0.4-0.9\,n_j$, which is at most $-0.5$ when $n_j\ge1$. Since the printed probability is an increasing function of the linear predictor, I have $p_{ti}<p_{ci}$ for every individual. The average of $p_{ti}-p_{ci}$ is then negative for all values of $\alpha$, $\beta$ and $\gamma_f$, and it cannot equal $0.007$. In the public file of the source study the clusters hold between 30 and 990 voters. A plausible explanation is that cluster size entered the fit in other units, such as thousands of voters, but the paper does not say so, and the scale cannot be recovered from the text.

Evidence a historical Lean attribution (Generated status: clause support not certified in this release)

The evidence is the derivation above, together with a Lean 4 proof of the same statement for the model as printed, which covers every finite nonempty population. Rounding does not change the sign: coefficients that round to $0.4$ and $-0.9$ are at most $0.45$ and $-0.85$, so that $\tau+\phi n_j\le-0.40$ when $n_j\ge1$. As a check of the scale only, I fitted the same model to all 85 clusters of the public file with cluster size in thousands. The fit gives $\tau=0.43$, $\beta=1.43$ and $\phi=-0.99$, with an average of $p_{ti}-p_{ci}$ equal to $0.010$. This is not the fit to 80 clusters that the paper reports, and it does not reproduce $0.007$.

What survives

The results on the estimators and their variances in Sections 2–5 and the appendices do not use the fitted coefficients. The scenario under the sharp null in Section 6.4 and Figures 2 and 3 uses a separate imputation and is unaffected. Section 6.5 and Figures 4 and 5 use the scenario with heterogeneous effects. No result there is shown to change, but the scenario cannot be reproduced from the model as printed. The units that were used and the coefficients of the fit to 80 clusters are not recovered.

The problem

As printed, the two covariances in (31) are the same, and the term that the next sentence calls the bias is identically zero. Subtracting equation (30) from equation (29), I obtain the treatment-arm mean $\overline{n_{tj}}$ in the second covariance, and Section 5.3 prints the formula in that form on p. 57. The second subscript $c$ is a misprint.

Evidence a derivation

The evidence is a derivation and an exact example. Let $\overline{n_{tj}}$ and $\overline{n_{cj}}$ be the mean sizes of the clusters assigned to treatment and to control in a given randomization, so that $\mathrm{E}[\overline{n_{tj}}]=\mathrm{E}[\overline{n_{cj}}]=N/M$. The estimator $\hat k$ is constant within a randomization. By equation (21) with $\hat k$ in place of $k$, I have

Taking expectations, I obtain

which is equation (29). By the same argument for the control arm, I have equation (30), $\mathrm{E}\bigl[\widehat{Y^T_{0,R1}}\bigr]=Y_0^T-M\operatorname{Cov}(\hat k,\overline{n_{cj}})$. Subtracting (30) from (29) and dividing by $N$, I obtain the corrected (31), since $\Delta=(Y_1^T-Y_0^T)/N$.

For the example, take $M=3$ clusters of sizes $(1,2,3)$, so that $N=6$, with one cluster treated. Let the cluster totals be $(0,0,0)$ under control and $(0,0,3)$ under treatment, so that $\Delta=1/2$, and let $\hat k$ be the least-squares slope of the observed cluster totals on $n_j$, with an intercept. Over the three assignments, $\hat k$ takes the values $(0,0,3/2)$ and the estimator takes the values $(0,0,3/8)$. Its expectation is $1/8$, and its bias is $-3/8$. Here $\operatorname{Cov}(\hat k,\overline{n_{cj}})=-1/4$ and $\operatorname{Cov}(\hat k,\overline{n_{tj}})=1/2$, and the bias term of the corrected formula is $(3/6)(-1/4-1/2)=-3/8$. The printed formula gives $0$.

What survives

Equations (29) and (30), their derivation and the bias formula of Section 5.3 are correct as printed. The unbiasedness of the Des Raj estimator for a coefficient $k$ fixed before the analysis is not affected. Only the last display of Appendix B changes.

The problem

Each item is a slip in a label, an index, a cross-reference or a bracket, and the neighbouring lines show the intended form. In items 6 and 12 a displayed line is false as printed.

1. $M$ is the number of clusters (p. 44), and the parallel construction on p. 46 says “$M$ clusters”.
2. Equation 9 is a probability limit. The estimator with an inner summation is defined in equation 6.
3. The display above the clause writes the control mean as $\overline{Y^T_{cj}}$.
4. Equations (20) and (21) estimate the population totals $Y^T_0$ and $Y^T_1$. The symbol $Y^T_{0j}$ is the control total of cluster $j$ (p. 48).
5. The superscript $T$ marks a total (p. 48). Individual outcomes do not carry it.
6. The sentence before the display defines $U^{T*}_{0j}=b_0\cdot N/M+b_1\cdot U^T_{0j}$ for the control clusters, and the third line of the display has $U^T_{0j}$ in the sum over $J_0$.
7. The individual covariate is $X_{aij}$. The same paragraph defines the total $X^T_{aj}=\sum_{i=1}^{n_j}X_{aij}$.
8. The regression is of the cluster total $Y^T_j$, as on p. 54.
9. The mean $\overline{n_{cj}}$ is taken over the clusters in the control condition. The clause for $\overline{n_{tj}}$ says “for clusters in the treatment condition”.
10. Equation 20 defines the estimator of the control total. The display that follows is the expectation of the estimator of the treatment total, equation 21.
11. Equation 32 has one variance for each arm, and the expansions that follow, “respectively”, are (33) for $\sigma^2(U^T_{j1})$, (34) for $\sigma^2(U^T_{j0})$ and (35) for the covariance.
12. The expectation is of the product of the two factors. The next line of the display expands that product inside the expectation.

Note that Appendix C writes the indices in the order $j0$, $j1$, and the main text writes $0j$, $1j$. This is a relabelling and needs no correction.

Evidence a derivation

The evidence is a check of the pages against the definitions on pp. 44, 46, 48 and 54, with a derivation for the two displayed lines. In equation (25), with $U^T_{0j}$ in the control sum, the constants $b_0\cdot N/M$ cancel between the two arm means, and $b_1$ times the original estimator remains. For example, let the cluster sizes be $(1,2)$, the control totals $(0,0)$ and the treatment totals $(1,2)$, with $k=0$, $b_0=0$ and $b_1=1$, and let the first cluster be treated. Then the estimate is $2/3$, and the second line as printed evaluates to $-2/3$. In the display that ends in equation (35), $\mathrm{E}$ is the average over the $M$ clusters and $\delta_j=n_j-N/M$ has average zero. Expanding $\mathrm{E}[(Y^T_{j0}-k\delta_j)(Y^T_{j1}-k\delta_j)]-\overline{Y^T_0}\,\overline{Y^T_1}$, I obtain $\sigma(Y^T_{j0},Y^T_{j1})-k\sigma(Y^T_{j0},n_j)-k\sigma(Y^T_{j1},n_j)+k^2\sigma^2(n_j)$, which is equation (35). With $k=0$ and both vectors of totals equal to $(0,2)$, the covariance is $1$. The line as printed instead equals $\mathrm{E}[Y^T_{j0}]\,Y^T_{j1}-1=Y^T_{j1}-1$, which is $-1$ for one cluster and $1$ for the other.

What survives

No result changes. Equation (10), the variance estimator on p. 51, the estimators (20) and (21), the invariance result (25), the bias formulas (29) and (30) and the optimal $k$ of Appendix C hold once the slips are corrected. The last lines of the displays (25) and (35) are correct as printed.

The problem

Every panel of Figures 2 and 4 carries its labels (Bias, SD and RMSE) in the upper left. Figure 5 shows the scenario with heterogeneous effects and displays all seven distributions of standard errors. Its last sentence repeats a remark from the caption of Figure 3. That figure concerns the sharp-null scenario, and the distributions for its three design-based estimators are omitted.

Evidence a check of the printed page

The evidence is a check of the pages: Figures 2 to 5 were inspected upright, panel by panel.

What survives

The panels and their numbers are unaffected. The sentence about the sharp null is true of Figure 3, whose caption already contains it. Naming the three labels in the corrected caption is optional, since “SE” is acceptable usage for the standard deviation of a point estimator.

The problem

The variances that enter equation (17), and the display that follows these lines on p. 52, are variances of cluster totals: $Y^T_{0j}$ and $Y^T_{1j}$ are defined on p. 48 and their variances on p. 50. Without the superscript the symbols $Y_{0j}$, $Y_{1j}$ and $Y_j$ are not defined. In $Y_0i$ the index $i$ is printed on the baseline. The symbols $n_c$ and $n_t$ denote the numbers of individuals in the control and treatment arms (p. 44), which are random under the assignment of clusters. The estimator $\hat{\mathrm{V}}(\widehat{\Delta_{HT}})$ has the denominators $m_c(m_c-1)$ and $m_t(m_t-1)$ and is undefined when $m_c$ or $m_t$ is $1$ (p. 51). Its stability therefore depends on the number of clusters in each arm.

Evidence a check of the printed page

The evidence is a check of the page against the definitions on pp. 44, 48, 50 and 51. The display that follows the two facts on p. 52 writes $\sigma^2(Y^T_j)$.

What survives

Equation (17), the display for $\mathrm{V}^N(\widehat{\Delta_{HT}})$ and the remark on stability are correct with these readings. Under the sharp null the identities hold for the cluster totals as they do for the individual outcomes. Nothing else changes.

The problem

Section 6.3 opens with “We will consider four regression-based estimators as well as the three design-based estimators proposed in this paper” (p. 61). It defines four regression estimators, and each of Figures 2–5 has four regression panels. The turnout covariate is defined on p. 59 as the “turnout rate in the cable district in the 2000 election”. The clusters are cable districts, and the turnout covariate is defined at that level.

Evidence a check of the printed page

I checked pp. 59–63 and the panels of Figures 2–5.

What survives

The estimators that are compared are those of Section 6.3 and the figures, and no number changes. The two covariates are indexed $X_2$, $X_3$ in the regressions (p. 62) and $X^T_1$, $X^T_2$ in the Des Raj model (p. 63). Each model defines its own indices, and this is not an error.

The problem

Section 6.3 specifies the standard errors of the two fixed-effects regressions (cluster-robust), of the random-effects regression (produced by lmer()) and of the three design-based estimators (the square root of the estimated variance under the sharp null). For the first regression, equation (27), it says only that the coefficient is fitted by ordinary least squares. Figures 3 and 5 nevertheless show a distribution of standard errors for this regression, and Section 6.4 interprets its upward bias. The paper does not say which standard error was computed, and the procedure cannot be recovered from the article or from the scripts of the source study, which serve a different analysis.

Evidence a recalculation from the source

The evidence is a check of the pages and a recalculation from the public data of the source study. Figure 3 reports a bias of 0.53 percentage points for this panel, measured from the standard deviation of the estimator, which Figure 2 reports as 2.4. Then the mean of the plotted standard errors is about 2.93 points. On whole-cluster subsets of the public data, the standard error from ordinary least squares cannot be that large. For a binary outcome regressed on a constant and a treatment indicator, the residual sum of squares is at most $N/4$, so that the residual variance satisfies $s^2\le N/(4(N-2))$. With $n_t$ and $n_c$ individuals in the two arms, the squared standard error of $\hat\beta_1$ is $s^2(1/n_t+1/n_c)$. Each arm consists of 40 whole clusters, and in the public data the 40 smallest of the 85 clusters contain $L=5{,}654$ individuals together, so that each arm has at least $L$ individuals. I have

With $N=22{,}733$ the square root of the right side gives a bound of about $0.767$ percentage points, for every assignment under these assumptions. This rules out ordinary least-squares standard errors for that sample construction. It does not identify the procedure or establish that the authors used the public data unchanged.

What survives

The point estimators as displayed and the labels printed in the figures stand. No standard-error implementation and no design-based formula is shown to be wrong. The distributions of the regression standard errors and the comparisons of their biases cannot be reproduced exactly.

Intended meaning

The passage warns that the difference in means is not unbiased in general when cluster size is related to outcomes. The warning is correct, and equation (8) and the paragraph on special cases state the exact condition. I write $R_1$ and $R_0$ for the mean outcomes of the treated and the control individuals under a given assignment, and $n_t$ and $n_c$ for the numbers of treated and control individuals. Then equation (8) reads

which implies that the bias is zero exactly when the two terms in the bracket are equal. The words “whenever” and “is biased” claim more than this. For example, let $M=2$ and $m_t=m_c=1$, and suppose that the first cluster has one individual with $Y_0=Y_1=0$ and the second has two individuals with $Y_0=Y_1=1$. The outcomes are perfectly correlated with cluster size, and both covariances equal $1/4$. The estimator takes the values $-1$ and $1$ with probability $1/2$ each, and its expectation is $0=\Delta$. In a second example the arms have unequal numbers of clusters ($M=3$, $m_t=1$, sizes $1$, $2$, $3$), and the effects are positive and vary with cluster size. There $\Delta=44/45$, each arm mean has bias $-7/90$, and the difference in means is unbiased. The balanced design under the sharp null in Section 6.4, for which the paper reports no bias, is another case of cancellation.

Note that equation (8) is correct as printed, including its sign, and that the estimator can be biased: with sizes $(1,2)$, control means $(0,1)$ and treatment means $(0,2)$ in the two clusters, its bias is $-1/6$. In the clause “no covariance between cluster size and outcomes”, the covariances are the two in equation (8). No estimator, derivation or reported number depends on the stronger wording.

Intended meaning

Section 5 concerns clusters of unequal size, and for these the formulas are correct. I write $\mathrm{V}(k)$ for the variance of the Des Raj estimator with a fixed $k$, so that $\mathrm{V}(0)$ is the variance of the Horvitz–Thompson estimator. Let $c=(M-m_c)/m_c$, $t=(M-m_t)/m_t$ and $v=(M-1)N^2/M^2$, as in Appendix C. By the expansion in Appendix C, I have

When $\sigma^2(n_j)>0$ the variance has a unique minimum at $k_{\mathit{optim}*}$, and the Des Raj estimator has the smaller variance exactly when $k(k-2k_{\mathit{optim}*})<0$. These are the strict intervals that the paper prints before the marked sentence. The two variances are equal at $k=0$ and at $k=2k_{\mathit{optim}*}$, and when $k_{\mathit{optim}*}=0$ no nonzero $k$ improves on the Horvitz–Thompson estimator. For example, let $M=4$ and $m_t=2$, with sizes $(1,2,3,4)$, control totals $(3,3,5,9)$ and a constant effect of $1$ on the totals. Then $k_{\mathit{optim}*}=2$, and the variance is $1.28$ at $k=0$, $0.2133$ at $k=2$ and $1.28$ at $k=4$.

When all clusters have the same size, $n_j-N/M=0$ for every $j$. The adjustment then vanishes, the Des Raj estimator coincides with the Horvitz–Thompson estimator for every $k$, and the quotients in equation (26) and in Appendix C are $0/0$. The Des Raj estimator remains unbiased in that case. Nothing in the paper depends on the two omissions.

Intended meaning

On p. 51 the paper calls a variance estimator conservative when its bias is nonnegative. The marked sentence restates that property for the sharp-null variance estimator: its expectation over the assignments is at least the true variance. The property holds. Let $d_j=Y^T_{1j}-Y^T_{0j}$ denote the effect on the total of cluster $j$, with mean $\overline{d}$ and variance $\sigma^2(d)$ over the $M$ clusters. Using equation (17) and the closed form of the estimator on p. 52, I obtain