Corrections to: On the Foundations of the Design-Based Approach

The problem

Equation (1) defines the raw potential outcome $y_i$ on assignment vectors $\mathbf z\in\mathcal Z$, and Equation (2) defines the exposure vector $\mathbf D=(g_1(\mathbf Z),\ldots,g_N(\mathbf Z))$ through exposure mappings that the researcher chooses. In the marked expression, the arguments of $y_i$ are an exposure $d$ for unit $i$ and the exposures $\mathbf D_{-i}$ of the other units. The paper does not define $y_i$ on such arguments. For the individualistic exposure mapping $g_i(\mathbf z)=z_i$, I have $\mathbf D=\mathbf Z$, and the expression is the average of the contrast in footnote 8 (p. 7), which holds fixed the treatments assigned to the other units. The sentence does not state this restriction, and the paragraph before it speaks of arbitrary exposure misspecification. For a general exposure mapping the comparison fails in two ways. First, the vector $(d,\mathbf D_{-i})$ need not lie in $\mathcal Z$ and need not be the exposure vector of any assignment. Further, two assignments with the same exposure vector can have different outcomes, so the exposures need not determine a unique outcome. Second, when $d$ and $d'$ are read as assignment coordinates and the expression has a value, the AEED of a coarsened exposure can differ from it by the same amount at every $N$, even if the assignments are independent and no outcome depends on another unit’s assignment. The comparison is correct for the individualistic exposure mapping under independent assignments, where the two targets are equal.

Evidence an exact counterexample

The evidence is two exact examples and a derivation. In the first example, let $N=2$ and $\mathcal Z=\{0,1\}^2$, let each assignment have probability $1/4$, and let the two units be adjacent in the network. The mapping of Equation (3) is then $g_1(\mathbf z)=z_1+2z_2$ and $g_2(\mathbf z)=z_2+2z_1$. Let $y_i(\mathbf z)=z_i$.

Each exposure has probability $1/4$ for each unit, and the AEED $\tau(1,0)$ equals $1$. Each value of $D_2$ occurs with exactly one value of $D_1$. For no assignment are both $(1,D_2)$ and $(0,D_2)$ exposure vectors, and the contrast for unit $1$ lacks a term in all four cases. The same holds for unit $2$. If $(d,D_2)$ is read as an assignment instead, it lies in $\{0,1\}^2$ only when $D_2\in\{0,1\}$, which has probability $1/2$. The same failure occurs for every even $N$ in a population of independent copies of this pair, although each exposure depends on one other exposure only and no outcome depends on another unit’s assignment.

In the second example, let $N\ge1$, let $Z_1,\ldots,Z_N$ be independent and uniform on $\{0,1,2\}$, and let $g_i(\mathbf z)=\mathbf 1\{z_i>0\}$, as in Section 6.2, and $y_i(\mathbf z)=z_i$. Then $\pi_i(1)=2/3$ and $\pi_i(0)=1/3$. By Equation (5), I have $\overline y_i(1)=(1+2)/2=3/2$ and $\overline y_i(0)=0$. Using Equation (8), I obtain $\tau(1,0)=3/2$ for every $N$. With $d=1$ and $d'=0$ read as assignment coordinates, the contrast $y_i(1;\cdot)-y_i(0;\cdot)$ equals $1$ for every unit and every draw, and the average direct effect is $1$. The two targets differ by $1/2$ at every $N$. The Horvitz–Thompson estimator of the AEED is $(3/(2N))\sum_{i=1}^NZ_i$, with expectation $3/2$ and variance $3/(2N)$, and it converges in probability to $3/2$. The exposures do not determine a unique outcome either. For $N=2$, the assignments $(1,0)$ and $(2,0)$ have the same exposure vector $(1,0)$, and the outcomes of unit $1$ are $1$ and $2$.

For the derivation, suppose that $g_i(\mathbf z)=z_i$, that $\pi_i(d)>0$ and $\pi_i(d')>0$, that $Z_i$ is independent of $\mathbf Z_{-i}$, and that $y_i(\mathbf Z)$ is integrable. For each $\mathbf w$ such that $\Pr[\mathbf Z_{-i}=\mathbf w]>0$, the assignment $(d;\mathbf w)$ has probability $\Pr[Z_i=d]\Pr[\mathbf Z_{-i}=\mathbf w]>0$, and hence it lies in the support of the design. By Equation (5), I find

In the second equality, I used that $Z_i$ is independent of $\mathbf Z_{-i}$. The same identity holds for $d'$. Subtracting the two identities and averaging over $i$, I obtain

The identity uses neither SUTVA nor the absence of interference. In an example with three units, independent binary assignments with probabilities $1/3$, $1/2$, $2/3$ and outcomes $y_i(\mathbf z)=z_i+2z_{i+1}+3z_iz_{i+1}$ with cyclic indices, both sides equal $5/2$. The second statement in the correction follows from the same computation with $\mathbf D$ in place of $\mathbf Z$ and $H_i$ in place of $y_i$.

What survives

The assignment space, the definitions of the EPO, AEPO, EED and AEED, the interpretation under NURVA and SUTVA, the examples and the arguments on external validity do not use this formula. The results on estimation and inference in the supplement do not use it either. No later result, table or figure depends on the sentence. Only the closing remark changes. The comparison with the average direct effect is restricted to the individualistic exposure mapping, or to exposure mappings for which the outcome is a function of the exposure vector. For dependent assignments a further condition is needed, which is the subject of P061-N009.

The problem

Section 3 fixes a finite population, real-valued outcomes and a probability measure that is a sum of point masses. It does not restrict the design to finitely many assignments, and it does not bound the outcomes. If the support of the design is countably infinite, then the sum in the numerator of Equation (5) need not converge absolutely, so a finite conditional mean is not guaranteed. Individual positivity controls only the denominator. In the generality of Section 3, positivity does not ensure that the EPO, the AEPO, the EED and the AEED exist, or that the Horvitz–Thompson estimator of supplement Equation (1) has an expectation. Every design in the article has finitely many assignments, and for those designs the targets are well-defined as stated.

Evidence an exact counterexample

The evidence is an exact counterexample that satisfies the setup of Section 3 and Condition 4.1. Let the population consist of one unit, and let $\Omega=\mathbb R$ and $\mathbf Z(\omega)=\omega$. Let $p(0)=1/2$, let $p(k)=2^{-k-1}$ for each integer $k\ge1$, and let $p(z)=0$ otherwise, so that the total mass is $1$. Let $g_1(z)=1$ if $z$ is a positive integer and $g_1(z)=0$ otherwise. Then $\pi_1(0)=\pi_1(1)=1/2$, and Condition 4.1 holds with $c=1/2$. Let $y_1(k)=(-1)^{k+1}2^{k+1}$ for each integer $k\ge1$ and $y_1(z)=0$ otherwise. Every outcome is a finite real number, and $y_1(k)p(k)=(-1)^{k+1}$.

Up to assignment $2m$, the positive terms of $\sum_k y_1(k)p(k)$ sum to $m$ and the negative terms sum to $-m$. Both parts diverge, and the partial sums alternate between $1$ and $0$. The numerator of Equation (5) at $d=1$ is undefined, and the denominator is $1/2$. The EPO at $d=0$ is $0$. Since $N=1$, the AEPO at $d=1$, the EED and the AEED of $d=1$ against $d=0$ are undefined as well. The Horvitz–Thompson estimator is $2Y_1\mathbf{1}\{D_1=1\}$, which is finite at every assignment and has no expectation.

The corrected statement holds by the following argument. Let

and suppose that $M_{i,d}<\infty$ and $\pi_i(d)>0$. Then $\operatorname{E}[|Y_i|\mid D_i=d]=M_{i,d}/\pi_i(d)<\infty$. The numerator of Equation (5) converges absolutely, and $\overline{y}_i(d)$ is the finite conditional expectation. Since $N$ is finite, Equations (6)–(8) are finite sums and differences of finite numbers. For the estimator, let $T_i=Y_i\mathbf{1}\{D_i=d\}/\pi_i(d)$. I have $\operatorname{E}|T_i|=M_{i,d}/\pi_i(d)<\infty$ and $\operatorname{E}[T_i]=\overline{y}_i(d)$, and linearity of expectation gives $\operatorname{E}[\widehat{\overline{y}}(d)]=\overline{y}(d)$. The condition $M_{i,d}<\infty$ holds whenever the support is finite. It also holds under NURVA at a targeted exposure: there $y_i$ takes a single value $a$ on the assignments in the support with exposure $d$, and $M_{i,d}=|a|\pi_i(d)$. For the bounded outcomes $y_1(k)=(-1)^{k+1}$ on the same design, I find $M_{1,1}=1/2$, and the EPO at $d=1$ and the expectation of the Horvitz–Thompson estimator both equal $1/3$.

What survives

Equations (5)–(8) are unchanged, and the Horvitz–Thompson estimator is unbiased for the AEPO whenever the targeted first moments are finite. The survey, experiment and sampling-treatment examples of the article and Tables 1–5 have finitely many assignments and are unaffected, as are the distinction between NURVA and SUTVA and the variance formulas under NURVA. Proposition 1 of the supplement assumes NURVA and the bound of Condition 1.1, which imply that the targeted first moments are finite. The added condition is needed in the statements on pp. 2, 6–7 and 14 that the targets are well-defined, in the unbiasedness derivation of supplement Equation (3), and for the AEPO and the Horvitz–Thompson expectation used in supplement Equation (10). The proof of supplement Equation (10) also needs the expectations of the predictions in it to be finite.

The problem

The citation places Sävje before Wang, and the entry in the reference list places Wang before Sävje. At least one of the two is misprinted. The article does not determine which order is that of the cited work.

Evidence a check of the printed page

I compared the author orders on the two pages. I did not compare either order with the cited work, so I have not established the correct order.

What survives

The cited work can be identified from either order. No argument in the article depends on the citation.

The problem

The article calls the targets of Section 4 new or novel in three places. Earlier work defines the same conditional mean and the same contrast on their shared domain, and the article does not say so. Sävje (2024) defines the conditional mean of Equation (5), with the expectation taken over the design, and the average contrast of Equation (8), for an arbitrary exposure mapping. When the exposure of a unit is its own binary treatment, $g_i(\mathbf z)=z_i$, the AEED is the average distributional shift effect (ADSE) of Sävje, Aronow, and Hudgens (2021). The article cites both works for other purposes, on p. 5 and in footnote 8, and footnote 7 credits Hudgens and Halloran (2008) with a special case of the EPO. The defect lies in the claim of novelty and in the missing attribution. Equations (5)–(8) are correct as definitions.

Evidence a historical Lean attribution (Generated status: clause support not certified in this release)

The evidence is a comparison with the earlier definitions and a derivation of the identities. I read Sävje (2024) in its author manuscript of 13 March 2023 (arXiv:2103.06471v2); a manuscript of the same paper dated 26 November 2019 has the same two definitions. I did not obtain the typeset text of the journal article. I read Definition 4 of Sävje, Aronow, and Hudgens (2021) in the author manuscript of the published article and in arXiv:1711.06399v4.

The identities hold for every design with finitely many assignments. By the definition of conditional probability, the ratio in Equation (5) equals $\operatorname{E}[y_i(\mathbf Z)\mid D_i=d]$ whenever $\pi_i(d)>0$. Substituting this identity into Equations (6)–(8), I obtain the contrast of Definition 1. If $g_i(\mathbf z)=z_i$ and the treatment is binary, then $D_i=Z_i$ and $Y_i=y_i(\mathbf Z)$, and the contrast $\tau(1,0)$ is the ADSE. The three identities are also proved in Lean, as identities between weighted sums over an arbitrary finite set of assignments.

What survives

Equations (5)–(8), the values of the targets, the analysis of NURVA and SUTVA, the examples, Tables 1–5, Figure 1 and the estimation results of the supplement do not depend on whether the targets are new, and none of them changes. I did not assess the novelty of the framework as a whole, of the survey-sampling interpretation or of NURVA. Footnotes 7 and 8 are not affected.

The problem

Equation (5) writes the EPO as $\overline y_i(d)$, and the paper reserves $y_i(\cdot)$ for raw potential outcomes of an assignment vector. The two quantities that this passage calls EPOs are printed without the overbar. In the same passage the argument of $g_i$ is an italic $z$ where the quantifier has the bold vector $\mathbf z$, and the first subscript $Z'$ is italic where the next paragraph has the bold $\mathbf Z'$.

Evidence a check of the printed page

The evidence is a check of p. 13 against Equation (5) on p. 6 and the vector notation on p. 4, including the fonts of the marked symbols.

What survives

The argument of Section 6.3 is unaffected. NURVA for each of two designs does not make their EPOs or AEEDs equal, and additional stability assumptions are needed.

The problem

The sentence cites Sävje (2024) for an equivalence between inference on AEEDs and inference on the average direct effect when the dependence in the design and the unmodeled interference are “sufficiently local”, and neither the paper nor its supplement says what that requires. I take the individualistic exposure mapping $g_i(\mathbf z)=z_i$, with feasible assignments and finite means defining the average direct effect (see P061-N019), and an estimator of the AEED with standard error $s_N$. The statistic standardized at the AEED and the statistic standardized at the average direct effect differ by $\delta_N/s_N$, where $\delta_N$ is the difference between the two targets. These statistics are asymptotically equal if and only if $\delta_N=o(s_N)$. It is not enough that $\delta_N$ tends to zero. In arXiv version 2 of Sävje (2024), including its supplement, the central limit theorem under local dependence is centred on the AEED, and I found no result there transferring inference to the average direct effect. Locality does not imply the rate. In the example below, each unit depends on one other unit only and the conditions of that theorem hold, while $\delta_N/s_N$ diverges. Conditions stronger than locality can make the printed sentence true, and the sentence is not shown to be false under every interpretation of “sufficiently local”.

Evidence an exact counterexample

The evidence is a derivation, an exact example, and a check of the cited works. For the derivation, let $\tau_N$ be the AEED, $A_N$ the average direct effect, $\delta_N=\tau_N-A_N$, and $\widehat\tau_N$ an estimator with standard error $s_N>0$. Both targets are deterministic, and

If $(\widehat\tau_N-\tau_N)/s_N$ converges in distribution to $W$ and $\widehat s_N/s_N\to1$ in probability, then Slutsky’s theorem implies that $(\widehat\tau_N-A_N)/\widehat s_N$ converges in distribution to $W$ when $\delta_N/s_N\to0$, and to $W+c$ when $\delta_N/s_N\to c\ne0$.

For the example, fix an integer $m\ge2$ and take $m^8$ disjoint pairs of units, so that $N=2m^8$. The pairs are assigned independently. Within a pair the assignments $(0,0)$ and $(1,1)$ each have probability $(1+1/m)/4$, and $(0,1)$ and $(1,0)$ each have probability $(1-1/m)/4$. Let $g_i(\mathbf z)=z_i$, and let $y_i(\mathbf z)=1+z_j/m$, where $j$ is the other unit in the pair of $i$. Each unit has $\Pr[D_i=1]=1/2$, the outcomes lie in $[1,3/2]$, and each assignment and each outcome depends on at most one other unit. The correlation of the two assignments in a pair is $1/m$, and the assignment of the other unit in the pair changes the outcome by $1/m$. The outcome does not depend on the unit’s own assignment, and hence $A_N=0$. Since $\Pr[D_j=1\mid D_i=1]=(1+1/m)/2$ and $\Pr[D_j=1\mid D_i=0]=(1-1/m)/2$, I have $\tau_N=1/m^2$ and $\delta_N=1/m^2\to0$. The Horvitz–Thompson estimator of the AEED is unbiased, with variance

A direct computation gives $\delta_N^2/s_N^2=4m^4/V_m\ge m^4/5$. The difference between the two targets is of order $N^{-1/4}$ and the standard error is of order $N^{-1/2}$. By Chebyshev’s inequality, the interval $\widehat\tau_N\pm5s_N$ covers the AEED with probability at least $24/25$, and for $m\ge5$ it covers the average direct effect with probability at most $V_m/m^4\le20/m^4$.

The example violates NURVA. It satisfies the conditions of the central limit theorem in the version of Sävje (2024) that was read, which is Proposition S8 of the supplement to arXiv version 2. The outcomes are bounded, the exposure probabilities equal $1/2$, and the variance is of order $N^{-1}$. Each unit depends on its own pair only, and hence Condition S6, $d_{\max}=o(n^{1/4})$, holds. Propositions S7 and S8 of that supplement keep the limit centred on the expected exposure effect. Lemma A20 of Sävje, Aronow, and Hudgens (2021), read in Supplement A of arXiv version 4, bounds the difference between the corresponding two targets of that paper by $6k^2n^{-1}\alpha_{\mathrm{int}}$ in its notation, and this bound tends to zero only when $\alpha_{\mathrm{int}}=o(n)$. Both works were read as preprints, and the published journal versions were not compared.

What survives

The definitions of the AEPO and the AEED, their interpretation, the examples, the finite-sample unbiasedness of the Horvitz–Thompson estimators, and the variance and consistency results stated under NURVA do not use this comparison. The closing remark changes. Using the same estimator and uncertainty scale for the average direct effect requires the rate condition below to preserve the first-order limit. Whether the journal version of Sävje (2024) contains a result of this kind was not checked.

The problem

Both rates are written in $n$, which the passage does not define. The supplement indexes its populations by their size $N$ and lets $N$ tend to infinity (p. 5), and on p. 1 it uses $n$ for the number of units sampled into exposure $d$. The rates refer to the number of units in the dependency graph, which is $N$.

Evidence a check of the printed page

The evidence is a check of pp. 1, 5 and 6 of the supplement.

What survives

Nothing depends on the symbol. The correction concerns the notation only. It does not assess whether the exponents are those of the cited works.

Intended meaning

The advice is to restrict the target population to the units that are eligible for the target, and it is correct whenever such units exist. For a finite set $T$ of target exposures, let $U_T=\{i:\pi_i(d)>0\text{ for every }d\in T\}$. If $U_T$ is nonempty, then $c=\min_{i\in U_T,\,d\in T}\pi_i(d)$ is positive because $U_T$ is finite. Then Condition 4.1 holds on $U_T$, and the targets are averages over $U_T$ with its size as denominator. If $U_T$ is empty, no trimmed population exists. For example, let there be two units with equally likely assignments $(0,0)$ and $(1,1)$, and let $g_1(\mathbf z)=z_1$ and $g_2(\mathbf z)=1+z_2$. Then each of the exposures $0$ and $2$ has probability $1/2$ for one unit and probability $0$ for the other. No definition or result in the paper uses the remark.

Intended meaning

The intended condition is that, for every unit and every pair of feasible assignments, equal exposures imply equal raw potential outcomes. Sections 5 and 6 use this condition, and it is correct. The EPO of Equation (5) is a conditional mean under the implemented design, defined only when $\pi_i(d)>0$, while Condition 5.2 ranges over all of $\mathcal Z$. For example, let $N=2$ and $\mathcal Z=\{0,1\}^2$, let the assignments $(0,0)$, $(0,1)$, $(1,0)$, $(1,1)$ have probabilities $1/2$, $1/4$, $1/4$, $0$, and let $g_i(\mathbf z)=z_1+z_2$ and $y_i(\mathbf z)=i(z_1+z_2)$. The raw potential outcomes satisfy the intended condition on $\mathcal Z$, and the exposures $0$ and $1$ each have probability $1/2$. Since $\pi_i(2)=0$, the last term of the display is $0/0$ at $\mathbf z=\mathbf z'=(1,1)$. No such gap occurs in Condition 5.1, and no supported EPO, table or later argument changes.

Intended meaning

On p. 8 the article says what SUTVA adds to NURVA. Under Condition 5.2, the outcome of a unit depends on the assignment only through the exposure of that unit, over the whole design space $\mathcal Z$. The EED can then be interpreted without the qualifiers about the design. NURVA requires this stability only on the support of the design. In that sense both sentences are correct: NURVA alone does not guarantee an off-support interpretation, and the interpretation without qualifiers holds exactly when Condition 5.2 does.

Read as statements about every off-support target, the words “only” and “required” say too much, and Section 5.2 (p. 9) contains an exception. I consider four units in the clusters $\{1,2\}$ and $\{3,4\}$, with the design space $\{0,1\}^4$, the exposure mapping $g_i(\mathbf z)=z_i$ and the outcomes $y_i(\mathbf z)=z_i+z_{m(i)}$, where $m(i)$ is the other unit in the cluster of $i$. Let the design put probability $1/2$ on each of $(1,1,0,0)$ and $(0,0,1,1)$. Individual positivity and NURVA hold. SUTVA fails, since $y_1(1,0,0,0)=1$ and $y_1(1,1,0,0)=2$ at the same exposure. The AEED is $\tau(1,0)=2$, and it equals the average effect of the assignment $(1,1,1,1)$ against $(0,0,0,0)$, two assignments outside the support. The equality holds for all outcomes of the form $y_i(\mathbf z)=F_i(z_i,z_{m(i)})$, because under this design $\overline{y}_i(d)=F_i(d,d)$. NURVA alone does not guarantee this equality. If the outcomes at $(1,1,1,1)$ are changed to $0$, then the distribution of $(\mathbf Z,\mathbf Y)$ under the design and the AEED are unchanged, and the average effect of $(1,1,1,1)$ against $(0,0,0,0)$ becomes $0$. No calculation that I inspected uses the stronger reading as a premise.