Corrections to: A Class of Unbiased Estimators of the Average Treatment Effect in Randomized Experiments

The problem

The set $\{Y_j:T_i\perp\!\!\!\perp T_j\}$ is defined by the independence of $T_i$ from each $T_j$ separately. A parameter $\theta_i$ fitted from several of these outcomes is a function of several assignments jointly, and $T_i$ can be independent of each of them and still depend on the vector they form. The conclusion $T_i\perp\!\!\!\perp\theta_i$ does not follow from the stated premise, and neither does $\operatorname{Cov}(T_i,f(\mathbf{X}_i,\theta_i))=0$. Footnote 5 treats a fit that uses one outcome $Y_j$, in which case $\theta_i$ is a function of $T_j$ alone, and for that case its argument is correct. The reduction to that case is not without loss of generality when the fit uses several outcomes.

When the premise holds only pairwise, the generalized difference estimator can be biased. In the design with six units described below, every randomization has three treated units and three controls, every outcome used in a fit belongs to the printed set, the estimate is $0$ in every randomization, and the average treatment effect is $1$.

Evidence an exact counterexample

This correction rests on an exact counterexample, computed in rational arithmetic, and on a derivation of the corrected condition.

Counterexample. Let $N=6$, and let the randomization $(T_1,\ldots,T_6)$ take the four values $010101$, $011010$, $100110$, $101001$ with probability $1/4$ each. Equivalently, $T=(A,1-A,B,1-B,C,1-C)$, where $A$ and $B$ are independent fair bits and $C=A+B-2AB$. Every unit has $\pi_{1i}=\pi_{0i}=1/2$. For two units in different pairs among $\{1,2\}$, $\{3,4\}$, $\{5,6\}$, each of the four joint outcomes of their assignments has probability $1/4$, and the two assignments are independent. The printed set then contains $Y_3,Y_4,Y_5,Y_6$ for units 1 and 2, $Y_1,Y_2,Y_5,Y_6$ for units 3 and 4, and $Y_1,Y_2,Y_3,Y_4$ for units 5 and 6. The assignment $T_5$ is not independent of the pair $(T_1,T_3)$: $\Pr(T_5=0,T_1=0,T_3=0)=1/4$, while $\Pr(T_5=0)\Pr(T_1=0,T_3=0)=1/8$.

Let $Y_{0i}=0$, $Y_{1i}=1$ and $X_i=1$ for every unit, and let $f(X,\theta)=X\theta$. Then $Y_i=T_i$ and $\Delta=1$. Let $Q(u,v)=u+v-2uv$, and fix the rules

Each rule uses two outcomes of the printed set for its unit and no other random variable. Since $Q(B,C)=A$, $Q(A,C)=B$ and $Q(A,B)=C$, I have $f(X_i,\theta_i)=\theta_i=T_i$ for every unit in every randomization. Hence $\operatorname{Cov}(T_i,f(X_i,\theta_i))=1/2-1/4=1/4$ for all $i$. Substituting $\pi_{1i}=\pi_{0i}=1/2$ and $f(X_i,\theta_i)=T_i$ into [13] and [14], I obtain

so that $\widehat{\Delta}_G=(3-3)/6=0$ in every randomization, and the bias is $-1$. By equation [19], the bias is again $-(1/6)\cdot 6\cdot(1/4)\cdot(2+2)=-1$. The Horvitz–Thompson estimate equals $1$ in every randomization.

Derivation of the corrected condition. Fix a set $J_i$ of units, and let $V_i=(T_j)_{j\in J_i}$. Suppose that $T_i$ is independent of $V_i$ and that $\theta_i$ depends only on $(Y_j)_{j\in J_i}$ and on fixed quantities. Since the potential outcomes are fixed, equation [1] implies that $f(\mathbf{X}_i,\theta_i)=H_i(V_i)$ for a fixed function $H_i$. For $a\in\{0,1\}$ and any set $D$ of values,

The sums are finite. Hence $T_i$ and $f(\mathbf{X}_i,\theta_i)$ are independent, and their covariance is zero. Equation [19] then implies that $\widehat{\Delta}_G$ is unbiased.

What survives

The Horvitz–Thompson estimator, the definitions [13]–[16], the bias identity [19], and the sufficient condition $\operatorname{Cov}(T_i,f(\mathbf{X}_i,\theta_i))=0$ with its proof in Appendix B are unaffected. The two fitting examples in Section 5.1 satisfy the corrected condition. These are designs in which units are assigned independently and the fit for unit $i$ leaves out unit $i$, and designs in which blocks are randomized independently and the fit leaves out the block of unit $i$, as in Section 7. Under the sharp null the observed outcomes are fixed, and so is any fit from them. A prediction that is additive in the outcomes with fixed coefficients, $f(\mathbf{X}_i,\theta_i)=a_i+\sum_jb_{ij}Y_j$, has zero covariance with $T_i$ under the printed pairwise rule, because $\operatorname{Cov}(T_i,Y_j)=(Y_{1j}-Y_{0j})\operatorname{Cov}(T_i,T_j)=0$. What changes is the general rule for an arbitrary fit: the stronger premise gives a valid guarantee of unbiasedness and excludes designs such as the one above.

The problem

For binary indicators, $\Pr(T'_k\ne T'_l)=\pi'_{1k0l}+\pi'_{0k1l}$. Since clusters $k$ and $l$ are distinct, at least one of the two probabilities on the right is positive, and not necessarily both. If cluster $k$ can be treated only when cluster $l$ is treated, then $\pi'_{1k0l}=0$ although $k\ne l$. The paper assumes that the marginal probabilities are positive. It does not assume that they are equal for the two clusters, in which case both probabilities would be positive.

Three statements rest on the deduction. First, the estimator [34] divides by $\pi'_{1k0l}$ and is undefined for such a pair. Second, equation [35] sets the corresponding cross term to zero through $\varepsilon_{1k0l}$ and adds nothing in its place. The variance of $\widehat{\Delta}_{HT}$ contains the term $2Y'_{1k}Y'_{0l}/N^2$ for that pair, and [35] neither estimates nor bounds it, so that [35] is not conservative in general. Third, under the sharp null with $\pi'_{1k1l}>0$ and $\pi'_{0k0l}>0$ for all $k,l$, [35] is not unbiased in general. Appendix E is affected in the same way, since the bound [43] contains the products $Y'_{1k}Y'_{0l}$ for such pairs and these cannot be estimated by weighting with inverse probabilities. The same holds for footnote 9, already for $f\equiv 0$, and for the statements of the abstract, of p. 137 and of the discussion that the variance estimation is conservative.

The bias of [35] is as follows. Let $x_k=Y'_{1k}$ and $z_k=Y'_{0k}$, and let $Z_{11}$, $Z_{00}$ and $Z_{10}$ be the sets of ordered pairs $(k,l)$ with $k\ne l$ such that $\pi'_{1k1l}=0$, $\pi'_{0k0l}=0$ and $\pi'_{1k0l}=0$, respectively. Then

The first three terms are nonnegative. The last can be negative when the outcomes are positive, and it can outweigh the others. In the design with six units described below the sharp null holds, the variance is $9/700$, and the expectation of [35] is $-269/6300$.

There are two ways to correct this, and each has a cost. The first is to add a term to the estimator, given below. The corrected estimator is conservative for every design with positive marginal probabilities, but under the sharp null with positive same-arm joint probabilities it is unbiased only when the cluster totals agree across every pair with $\pi'_{1k0l}=0$. The second is to keep [34] and [35] and to assume that $\pi'_{1k0l}>0$ for all $k\ne l$. The printed statements then hold, for a smaller class of designs that excludes the one below.

Evidence an exact counterexample

This correction rests on an exact counterexample, computed in rational arithmetic, and on a derivation of the bias identity and of the corrected estimator.

Counterexample. Let $N=M=6$, with each unit its own cluster, and let all twelve potential outcomes equal $1$, so that the sharp null holds. Let the randomization be drawn uniformly from the fourteen randomizations such that exactly three units are treated and $T'_1\le T'_2$. The marginal probabilities of treatment are $(2/7,5/7,1/2,1/2,1/2,1/2)$. Every joint probability $\pi'_{1k1l}$ and $\pi'_{0k0l}$ is positive, the smallest being $1/14$, and every two indicators differ with probability at least $3/7$. The probability $\Pr(T'_1=1,T'_2=0)$ is zero, and it is the only $\pi'_{1k0l}$ with $k\ne l$ that vanishes. All hypotheses of the printed statements hold. Enumerating the fourteen randomizations, I find

a bias of $-1/18=-2\cdot 1\cdot 1/6^2$, in agreement with the identity above. The covariance of the two estimated totals is $27/175$, and [34] with its undefined term omitted has expectation $202/175$, which exceeds the covariance by $1$. With the added term the expectation of the variance estimator is $9/700$.

The corrected estimator is not unbiased under the sharp null in general. In the same design with both potential outcomes equal to $(1,2,1,1,1,1)$, the variance is $2759/12600$, the expectation of [35] as printed is $151/1400$, and the expectation of the corrected estimator is $3109/12600$. Its bias is $1/36=(1-2)^2/6^2$. Under complete randomization of three of six units every $\pi'_{1k0l}$ with $k\ne l$ equals $3/10$, and with sharp-null outcomes $(1,2,3,-1,0,4)$ the variance and the expectation of [35] both equal $7/3$.

Derivation. I take expectations in $N^2$ times [35] term by term and compare with $N^2\operatorname{Var}(\widehat{\Delta}_{HT})$, whose off-diagonal part is the sum over ordered pairs $k\ne l$ of

The diagonal terms of [35] have expectation $x_k^2/\pi'_{1k}+z_k^2/\pi'_{0k}$, which exceeds the diagonal part of the variance by $(x_k-z_k)^2$. A product term with a positive joint probability is unbiased for the corresponding term of the variance. For $(k,l)\in Z_{11}$ the variance has the term $-x_kx_l$, while [35] has a product term equal to zero and a bounding term with expectation $(x_k^2+x_l^2)/2$. The difference is $(x_k+x_l)^2/2$, and the same holds for $Z_{00}$ with $z$ in place of $x$. For $(k,l)\in Z_{10}$ the variance has the term $+2x_kz_l$ and [35] has zero. Summing these contributions, I obtain the identity stated above. The added term has expectation $N^{-2}\sum_{Z_{10}}(x_k^2+z_l^2)$, and $x_k^2+z_l^2-2x_kz_l=(x_k-z_l)^2$. For the corrected estimator the last term of the identity becomes $+\sum_{Z_{10}}(x_k-z_l)^2$, and every term is nonnegative. Under the sharp null, with $x_k=z_k=Y'_k$ and $Z_{11}$, $Z_{00}$ empty, the bias reduces to $N^{-2}\sum_{Z_{10}}(Y'_k-Y'_l)^2$.

What survives

The population variance and covariance identities [20]–[30] are unaffected by this error. In particular [30] is correct, because a pair $k\ne l$ with $\pi'_{1k0l}=0$ contributes $-Y'_{1k}Y'_{0l}$ to either form of the covariance. Only the sentence that justifies it changes. The Young bound [43] and the estimators [32] and [33] of the variances of the two estimated totals are unaffected, and so are the point estimators and their unbiasedness. In every design with $\pi'_{1k0l}>0$ for all $k\ne l$, the covariance and HT variance guarantees of [34] and [35] hold. The example of Section 7 is such a design: its within-block probabilities $\pi'_{1k0l}$ are $1/3$ and $4/15$ and its cross-block probabilities are $1/3$ and $1/6$, and the added term is zero there. Three things change. The estimator [35] gains a term in designs where one cluster can be treated only if another is. The statement that the variance estimator is conservative for any design holds for the corrected estimator and not for [35] as printed. Exact unbiasedness under the sharp null requires a further condition.

The problem

Table 2 reports, for the population of Table 1 and the estimators of Section 7.1, the moments of each estimator over the 90 possible randomizations. When they are recomputed from the printed population, three of the four columns that use the covariate $X$ differ from the table. For G (Linear) the standard error is $.246$ and the variance $.060$, where the table has $.162$ and $.026$. For G (Logit) the standard error is $.247$, and for FE (Cov.) it is $.279$. The variances differ by a factor of about two, which rounding cannot explain. All 40 entries of the four columns that do not use $X$ (IPW, FE, HT and G (Prior)) agree with the table to the third decimal.