Corrections to: Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence

The problem

Appendix C says that the treatment-list response of a design-affected subject is the control-list response plus one, capped at four. I call this the described rule. I write $W$ for the control-list response, and I call a respondent who answers “Yes” to the direct question an admitter, as Figure A1 does. In the simulation for the lower-left panel of Figure A1, no admitter confesses falsely or lies on the list. For a design-affected admitter, the response under the described rule is $\min(W+1,4)$, that is, $1,2,3,4,4$ for $W=0,\dots,4$. The replication code does something else. The function design_affector raises the count of non-sensitive items by one unless it is already four, and the simulator then adds the sensitive item. I call this the coded rule. Under the coded rule the response is therefore $\min(W+1,4)+1$, that is, $2,3,4,5,5$. The two rules differ at every value of $W$.

The two rules lead to different power. Placebo Test I tests whether the difference $\beta$ between the mean responses of the two arms among admitters equals 1. Under the described rule a design-affected admitter differs from an unaffected one only when $W=4$, an event of probability $0.4^4=0.0256$. Then $\beta$ falls below 1 by $0.0256$ times the affected share. Under the coded rule $\beta$ rises above 1 by $0.9744$ times that share. The lower-left panel of Figure A1 and the last clause of the sentence on 80% power report high power. The coded rule has it and the described rule does not. With 800 admitters and 20% design-affected, the power is about 0.78 under the coded rule and about 0.05, the level of the test, under the described rule. Footnote 1 says that power depends on the form of the design effect. The statement of 80% power is then read as a statement about the rule just described. The evidence cannot decide which of the two the authors intended, the description or the code.

Evidence an exact counterexample

The evidence is an exact comparison of the two rules and a simulation of both with the authors' test function. Running the function design_affector unchanged and adding the sensitive item, I obtain the following responses.

The control-list response $W$ is binomial with four trials and success probability $2/5$, and the values $0,\dots,4$ have probabilities $81/625$, $216/625$, $216/625$, $96/625$ and $16/625$. Let $A$ indicate a design-affected subject, and let $\rho$ be the share of such subjects, who are drawn at random independently of $W$ and of the treatment. The treatment-list response of an admitter is $W+1-A\,\mathbf{1}\{W=4\}$ under the described rule and $W+1+A\,\mathbf{1}\{W<4\}$ under the coded rule. Taking expectations, I obtain

At $\rho=1/5$ these values are $3109/3125=0.99488$ and $3734/3125=1.19488$.

The simulation follows the script for the panel: the stated number of admitters, 200 withholders and 200 respondents who do not engage in the behavior, with the specified share of all subjects designated design-affected. Both rules are applied to the same draws, the test is the authors' placeboI_test, and the null is rejected when $p<0.05$.

In the last two rows the coded rule led to rejection in all 20,000 experiments. In a second simulation, written independently and run with another seed, the power is 0.05145 and 0.77780 at 800 admitters and a share of 0.20. The full grid of Figure A1 was not computed again, and the values of its cells were not recovered. The simulation does not show that the power under the described rule stays near 0.05 in arbitrarily large samples, since $\beta_{\text{described}}$ differs from 1 whenever $\rho>0$.

What survives

The defect is confined to three places: the description of the design effect in Appendix C, the reading of the lower-left panel of Figure A1 as that design effect, and the last clause of the sentence on 80% power. The example in the main text, with about 80 percent power when 20 percent of 800 are false confessors, concerns false confessions and is not affected. The test formula of Proposition 3, the identification and efficiency results, the other panels of Figure A1, Placebo Test II, and the estimates and placebo p-values of the application do not depend on the simulated design effect. The high-power conclusion does not hold for the printed ceiling rule. Either the statement of 80% power is attached to the positive design effect that was simulated, or the ceiling rule is kept and the panel shows power near the level of the test at the point that the text cites.

The problem

The identity in the first sentence says that, among respondents who answer “Yes”, the list response in the treated arm is distributed as the list response in the control arm plus one. The distributions that are equal under the null are therefore those of $V_i$ in the control arm and $V_i-1$ in the treated arm, or equivalently those of $V_i+1$ in the control arm and $V_i$ in the treated arm. The second sentence compares $V_i$ in the control arm with $V_i+1$ in the treated arm, that is, $W_i$ with $W_i+2$. These two distributions need not be equal when all four assumptions hold, and a test built on the sentence as printed can systematically reject a valid model.

Evidence an exact counterexample

The evidence is a derivation and an exact counterexample to the printed comparison. Suppose that Assumptions 1–3 hold and that $Y_i=1$. Then $X_i=1$ by Assumption 1, and by Assumption 2 I have $V_i=W_i$ in the control arm and $V_i=W_i+1$ in the treated arm. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms. I conclude that $V_i$ in the control arm and $V_i-1$ in the treated arm have the same distribution. For the counterexample, let $X_i=Y_i$ be a Bernoulli variable with mean $1/2$, let $W_i$ and $Z_i$ be Bernoulli variables with mean $1/2$ such that $X_i$, $W_i$ and $Z_i$ are independent, and set $V_i=W_i+X_iZ_i$. All four assumptions hold. Among respondents with $Y_i=1$, the response $V_i$ in the control arm takes the values $0$ and $1$ with probability $1/2$ each. In the treated arm, $V_i+1$ takes the values $2$ and $3$ with probability $1/2$ each, and $V_i-1$ takes the values $0$ and $1$ with probability $1/2$ each.

What survives

The probability identity, the null $\beta=1$, the test of Proposition 3 and the reported placebo tests are unaffected. The replication code tests the difference in means against 1, as the paper describes. Only a distributional test implemented from the sentence as printed would use the wrong comparison.

The problem

Proposition 4 defines the p-value of Placebo Test II as two-sided. Figure A2 presents the power of the test at four sample sizes, computed with the R function power.prop.test for two groups of size $N/2$. By default this function counts, in a two-sided calculation, only the rejections in the direction of the true difference. The option strict=TRUE adds the rejections in the opposite direction. The calls for $N=200$, $500$ and $1000$ set the option, and the call for $N=100$ does not. The first panel therefore shows a different quantity from the other three. The omitted probability is positive when $S>0$, with $S$ defined below, and it is at most 0.025, the value that it takes when the two proportions are equal and lie strictly between 0 and 1. On that diagonal the first panel plots 0.025 where the other panels plot 0.05.

Evidence an exact counterexample

The evidence is a derivation of the omitted term, its exact value at one plotted point, and an evaluation of the original calls at every plotted point. Let each group have size $n$, let $p_1$ and $p_2$ be the two proportions, and set $d=|p_1-p_2|$, $S=p_1(1-p_1)+p_2(1-p_2)$, $V=(p_1+p_2)(1-(p_1+p_2)/2)$ and $c=\Phi^{-1}(0.975)$. Suppose that $S>0$. In the normal approximation of the function, the probability of rejection in the direction of the true difference is $A$ and the probability of rejection in the opposite direction is $B$, where

Line 31 plots $A$, and the other three panels plot $A+B$. Since $V-S=(p_1-p_2)^2/2\ge0$, I have $(\sqrt n\,d+c\sqrt V)/\sqrt S\ge c$, which implies that

with equality when $p_1=p_2$. At $n=50$ and $p_1=p_2=1/2$ the plotted value is exactly $1/40$, and the value with both tails is $1/20$.

Among the 10,201 points of the first panel, the largest change at a point whose plotted power is at least 0.74 is $1.3\times10^{-6}$. No point enters or leaves the red band that marks power near 0.80. The two corner points at which both proportions are 0 or both are 1 have $S=0$ and are undefined in every panel.

What survives

The other three panels, the red band near 0.80 on the plotted grid, and the definition of the test in Proposition 4 are unaffected. No empirical p-value uses this function: the replication code computes the p-values of Placebo Test II directly as $2\Phi(-|\hat\delta|/\sqrt{\widehat{\operatorname{Var}}[\hat\delta]})$ in a separate function. The correction stays within the normal approximation and does not make the figure an exact calculation of power in finite samples.

The problem

The readme says that the archive contains the programs needed to replicate the figures and tables. The two power scripts and the two analysis scripts each begin with a call that sets the working directory, and a comment asks the user to set it. The readme describes a layout with folders for programs and data. This is ordinary setup. After the setup, the script for Figure A1 still cannot run. Its four calls to the simulator use the name listsimulator, which is not defined, and argument names with dots. The function supplied is list_simulator, and its argument names have underscores. The supplied function itself stops on every call that reaches line 112, since placeboI_test returns a named numeric vector and $p_value cannot be extracted from such a vector. The script for Tables A2–A8 stops at its first call to xtable, a package that it never loads. The script for Figure A1 also sets no seed, and the Monte Carlo draws of the published figure cannot be repeated.

Evidence an exact counterexample

The evidence is a set of executable checks on the unchanged files, each in a new R session. The four calls to the simulator report could not find function "listsimulator". With that name pointed at list_simulator, they report unused arguments. A direct call of list_simulator reports $ operator is invalid for atomic vectors. A small example makes the last failure exact. For $V=(0,2,1,3)$, $Z=(0,0,1,1)$ and $Y=(1,1,1,1)$, the function placeboI_test returns a numeric vector with estimate $1$, standard error $\sqrt2$, p-value $1$ and $n=4$, and $p_value fails on it. With the directories set, the script for Tables A2–A8 stops at line 45 with could not find function "xtable". With the directories set and xtable made available, and with the estimator code unchanged, I find that all 360 numeric entries of Tables 3–8 and of Tables A1–A8 of the supplement agree with the publication at the printed precision.

What survives

These faults do not show that any theorem, estimator formula, estimate or power value is wrong. The functions that produce the empirical tables do not call the simulator, and the 360 table entries compared here agree. Two points are not established: which version of the code drew the published Figure A1, and whether the full grid of that figure is recovered once the script runs. The full grid was not computed again.

The problem

The paper’s framework has one behaviour $X_i$ for each topic: the list counts it for treated respondents, and the direct question asks about it. For CNN the two questions do not ask about the same behaviour. Under the ordinary reading of “read or watch”, a respondent counts CNN in the list if she reads it or watches it, and she answers “Yes” to the direct question only if she watches it. A respondent who reads CNN and does not watch it answers both questions truthfully when she counts CNN in the list and answers “No”. Therefore, for CNN, a gap between the list-based estimates and the direct estimate need not reflect withholding, and the standard list and combined estimates may measure reading or watching CNN, which is a broader behaviour than the one in the direct question. The remark in Section 5.2 is tentative, but for CNN the wording of the questions does not support it. The paper does not show how respondents understood the list question, and some may have taken “read or watch” to mean reading newspapers and watching broadcasters.

Evidence an exact counterexample

The evidence is a check of the page and an exact example. Suppose that one-quarter of the population watches CNN and says so, one-quarter reads CNN without watching it, and one-half does neither. Everyone answers both questions truthfully, the baseline count is 1 or 2 with equal probability, and assignment is by a fair coin independent of everything else. Then the mean direct answer is $1/4$, which is the share that watches CNN, while the standard list difference and the combined expression (1) both equal $1/2$, the share that reads or watches. With $X_i$ taken as reading or watching, the numerical conditions of Assumptions 1–4 hold, and so do the null hypotheses of both placebo tests ($\beta=1$ and $\delta=0$). If those who only read CNN are replaced by watchers who answer “No”, then the joint distribution of assignment, direct answer and list count is unchanged, and the responses cannot distinguish the two cases. If those who only read CNN are 1 percent of the population, then the mean direct answer is $1/4$ and the two list-based quantities are $13/50$. Near agreement of the estimates does not show that the questions measure the same behaviour. The example shows what the wording allows. It does not show that such respondents were in the sample, and no reported CNN estimate is shown to be wrong.

What survives

The identification and efficiency results, which assume a single behaviour $X_i$, are unaffected, and so are the other four topics. The CNN entries of Tables 3 and 6 and of Figures 1 and 2, the placebo tests and the replicated effect of the treatment list on direct answers about CNN stand as statements about the responses. Their interpretation for CNN changes in the tables and figures, in the replication, and in supplement Appendix E, which describes the standard list estimate as an estimate of “those watching CNN”. The paper already reports that CNN fails Placebo Test I in Study A ($p=0.034$) and Placebo Test II in Study B ($p=0.003$), and it calls the combined estimate for CNN in Study B unreliable.

The problem

(a) The middle expression of equation (2) is a difference of two expectations. It should contain the mean $\mathrm{E}[W_i]$ of the baseline response and not the random variable $W_i$. The two terms cancel in either form, and the value $\mu$ is unaffected. (b) The equality $\mathrm{E}[X_i\mid Y_i=1]=1$ says that no respondent confesses falsely, which is Monotonicity (Assumption 1). Assumption 2 is No Liars and No Design Effects, and the sentence after the proof calls Assumption 1 Monotonicity. (c) The $\mu$ before the coverage percentage does not belong to the sentence. (d) Display (4) conditions on $Y_j=0$ for $j=1,\dots,k$, and the paragraph begins “Suppose that exactly $k$ direct responses are zero”. The reordered responses are therefore the zeros. (e) The step equates the baseline means of the two arms among respondents with $Y_i=1$, which uses Treatment Independence (Assumption 3) as well as Assumptions 1 and 2. All three are hypotheses of Proposition 2.

Evidence a derivation

The evidence is a check of the pages against the definitions of the assumptions on pp. 46–47, and a derivation of the identities in (a) and (e). For (a), recall that Assumption 2 states that $V_i=W_i+X_iZ_i$. By Assumption 3, the pair $(W_i,X_i)$ is independent of $Z_i$, and I have

For (e), suppose that $Y_i=1$. Then Assumption 1 implies that $X_i=1$, and by Assumption 2 I have $V_i=W_i+Z_i$. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms, so that $\mathrm{E}[W_i\mid Z_i=1,Y_i=1]=\mathrm{E}[W_i\mid Z_i=0,Y_i=1]$. I conclude that $\mathrm{E}[V_i\mid Z_i=1,Y_i=1]=\mathrm{E}[V_i\mid Z_i=0,Y_i=1]+1$.

What survives

Nothing depends on these slips. The identity (1) of Lemma 1, the coverage statement of Corollary 1 and the inequality of Proposition 2 are unchanged, and so are the estimator and the reported analyses. Other steps of the proofs in Appendix B are the subject of P019-E003.

The problem

Sections 4.1 and 4.2 define two statistics. Placebo Test I uses $\hat\beta$, the difference in mean list counts between treatment and control among those who answer “Yes” to the direct question, with null value $\beta=1$. Placebo Test II uses $\hat\delta$, the difference between treatment and control in the proportion who answer “Yes”, with null value $\delta=0$. Tables 5, 8, A4 and A7 report Placebo Test II and head the estimate column with the symbol of Placebo Test I. Table A4 also carries the caption of Table A3. The word “direction” is a slip for “direct”. The citation of Dalgaard (2008) has no entry in the reference list on pp. 65–66, and the supplement has no reference list. In the scripts the label of the p-value column of Placebo Test I names the null $\beta=0$, while the function computes the two-sided p-value for $\beta=1$, as the paper states. Further, the label is written with a single backslash before beta, which R reads together with the letter b as a backspace character.

Evidence a check of the printed page

The evidence is a check of the pages and a recalculation from the replication files. All 80 entries of the four tables (estimates, standard errors, p-values and sample sizes) agree with the calculation of Placebo Test II at the printed precision, and each of the 20 estimates differs from the corresponding $\hat\beta$. For nuclear power in Study A, for example, $\hat\delta=276/4165=0.066$, the first entry of Table 5, while $\hat\beta=28321/26871=1.054$, the first entry of Table 4. I could not determine from the article, the supplement or the replication files which work the Dalgaard citation refers to, or check its page number.

What survives

Every estimate, standard error, p-value and sample size in the placebo tables stands, and so do the conclusions drawn from them, including the significant results of Placebo Test II for nuclear power and CNN in Study B. The corrections change headers, one caption, one word, one reference and a label in the code.

The problem

Monotonicity is Assumption 1, No Liars and No Design Effects together are Assumption 2, and Treatment Independence is Assumption 3. Assumption 4 is Non-Degenerate Distributions, which the second sentence of Proposition 3 invokes separately. The range that matches the four names is 1–3. The first paragraph of Section 4.1 lists the same four names, and Appendix C of the supplement calls the test “a joint test of Assumptions 1-3”. The printed implication is not false, since $\beta=1$ also holds under the stronger hypothesis. The misprint is in the label of the null hypothesis.

Evidence a derivation

The evidence is a check of the two passages against the definitions of the assumptions on pp. 46–47 and against p. 2 of the supplement, and a derivation of $\beta=1$ from Assumptions 1–3. Among respondents with $Y_i=1$, Assumption 1 implies that $X_i=1$, Assumption 2 then gives $V_i=W_i+Z_i$, and by Assumption 3 the mean of $W_i$ is the same in the two arms.

What survives

Nothing depends on the range. The identity $\beta=1$ and the p-value for large samples are unchanged. Assumption 4 is still needed, as the proposition says, so that $\beta$ is defined and the normal approximation applies.

The problem

Fisher’s method combines the five p-values $p_1,\dots,p_5$ of Table 7 into $T=-2\sum_{j=1}^{5}\log p_j$ and refers $T$ to the chi-square distribution with 10 degrees of freedom. This reference distribution is correct when the five p-values are independent, which here requires the five placebo statistics $\hat\beta_j$ to be asymptotically independent. The paper does not state this premise, and Assumptions 1–4 do not supply it. Each assumption concerns one list experiment and its direct question. In particular, Treatment Independence makes $(W_i,X_i,Y_i)$ for a topic independent of the assignment $Z_i$ to the list on that topic, and it says nothing about the assignments to the other four lists. In Study B each respondent answered all five lists and then the five direct questions. Therefore the assignment to one list may affect the baseline response to a later list, or a later direct answer, without violating any of the assumptions as stated; Section 5.3 itself raises the possibility of such priming. Note that the calibration does not fail merely because the same respondents enter all five tests. If the five assignments are mutually independent and jointly independent of the full vector $(W_j,X_j,Y_j)_{j=1}^{5}$, then the five statistics are asymptotically independent. The paper states no such joint condition.

Evidence an exact counterexample

The evidence is a derivation, an exact counterexample and a recalculation.

Derivation. I index the lists by $j$ and write $q_j=\Pr(Y_j=1)$, $\gamma_j=\Pr(Z_j=1)$ and $m_j=E[W_j\mid Y_j=1]$. Under Assumptions 1–4 for list $j$, I have $\beta_j=1$ and

The five statistics are jointly asymptotically normal with covariances $E[\psi_j\psi_k]$. Suppose that $Z_1,\dots,Z_5$ are mutually independent and that $(Z_1,\dots,Z_5)$ is independent of $L=(W_j,X_j,Y_j)_{j=1}^{5}$. For $j\neq k$, the conditional expectation $E[\psi_j\psi_k\mid L]$ contains the factor $E[Z_j-\gamma_j]\,E[Z_k-\gamma_k]=0$. Then the limiting covariance matrix is diagonal, the five p-values are asymptotically independent and uniform, and $T$ converges in distribution to $\chi^2_{10}$.

Counterexample. For each respondent, let $D\in\{0,1\}$, $U\in\{-1,1\}$ and $Z_1,\dots,Z_5\in\{0,1\}$ be independent, each taking its two values with equal probability. Let $A_j=2Z_j-1$ and define, with empty products equal to 1,

The baseline response to list $j$ depends on the assignments to the earlier lists only, and the direct answers, which follow all the lists, depend on the assignments to the later lists; the order of Study B permits both. Every list satisfies Assumptions 1–4 with respect to its own assignment, and $\beta_j=1$ for every $j$. Enumerating the 128 equally likely outcomes in exact arithmetic, I find that the covariance matrix of $(\psi_1,\dots,\psi_5)$ is

The standardized statistics have correlation $1/2$ among the first four lists and $1/\sqrt2$ between each of these and the fifth, while each individual p-value remains valid. In this limit the variance of $T$ exceeds 20, and a $\chi^2_{10}$ variable has variance exactly 20. Hence $T$ does not converge to $\chi^2_{10}$. The example is artificial. It shows that the premise does not follow from the stated assumptions, and it does not show that such dependence is present in the data.

Recalculation. From the replication files, I recover the entries of Table 7. The two joint tests are as follows.

The Wald statistic is $W=(\hat\beta-\mathbf 1)^{\top}\widehat C^{-1}(\hat\beta-\mathbf 1)$, where $\widehat C$ is the covariance matrix of $\hat\beta$ estimated from the respondents' influence values, so that covariance between lists is allowed. The estimated correlations between lists lie between $-0.034$ and $0.069$.

What survives

The entries of Table 7, the five individual tests, the value of Fisher’s statistic and its nonrejection all stand, and a joint test that allows dependence between the lists does not reject either. No other result of the paper uses this step. The joint p-value from Fisher’s method rests on a premise beyond Assumptions 1–4, and the paper’s own discussion of priming and order effects shows that the design does not guarantee this premise.

The problem

The sentence presents two causes as the only ones: false confessions, which violate Assumption 1, or a failure of No Liars or No Design Effects, which together are Assumption 2. The joint null of the test also contains Treatment Independence (Assumption 3), as the first sentence of the paragraph and Proposition 3 say. A failure of Assumption 3 alone can make the population difference $\beta$ differ from 1. This happens when the answer to the direct question depends on the list treatment. Then the respondents who answer “Yes” in the two arms differ in their baseline list responses, although no one confesses falsely, no one lies on the list, and the responses to the non-sensitive items do not change. Note that an effect of the treatment on the direct answer is not a design effect in the sense of Assumption 2; the paper places it under Assumption 3 (p. 47). Here I read “estimate” as the population quantity $\beta$, so that sampling variation is not at issue.

Evidence an exact counterexample

The evidence is a derivation and an exact counterexample. Under Assumptions 1 and 2, every respondent with $Y_i=1$ has $X_i=1$ and $V_i=W_i+Z_i$, and I have

Assumption 3 implies that the last two terms are equal, and without it they need not be. For the counterexample, let $B$, $C$, $X$ and $Z$ be independent Bernoulli variables with mean $1/2$, and set

Since $Y\le X$, Assumption 1 holds, and the three classes of respondents $(X,Y)=(0,0)$, $(1,0)$ and $(1,1)$ have probabilities $1/2$, $1/4$ and $1/4$. Assumption 2 holds by the definition of $V$. The treatment $Z$ is independent of $(B,C,X)$, but the direct answer depends on $Z$, and Assumption 3 fails: the event $(W,X,Y)=(0,1,0)$ has probability $1/16$ and never occurs together with $Z=0$. The list response $V$ has the following moments in the four cells.

Assumption 4 holds, since all four variances are positive, $\operatorname{Var}[Y]=3/16$ and $\operatorname{Var}[Z]=1/4$. From the two rows with $Y=1$, I obtain $\beta=5/2-1/2=2$. The share of “Yes” answers is $1/4$ in each arm, so that $\delta=0$. Equal response rates in the two arms therefore do not imply Assumption 3.

What survives

Section 4.1 and Proposition 3 include Treatment Independence in the null, and the implication $\beta=1$ under the stated assumptions is unaffected. In the simulations of Appendix C, the direct answers are fixed before the treatment is assigned at random. Assumption 3 holds there, and the omitted cause does not arise. The power values were not computed again for this finding. The application already separates Study A, in which the direct questions come first, from Study B, for which the paper discusses failures of Treatment Independence. What changes is the diagnosis. A rejection by Placebo Test I cannot in general be attributed only to false confessions or to the list responses.

Intended meaning

The proposition is stated within the model of Section 2, in which Assumptions 1–3 are maintained, and Assumption 4 is the condition added for inference. The sentence after the proposition, Corollary 1 and Proposition 2 all say “Assumptions 1–4”, and under these four assumptions formula (3) is correct. Read as a self-contained statement, the proposition omits hypotheses that (3) needs. To see where they enter, I write $q=\mathrm{E}[Y_i]$, $m_z=\mathrm{E}[V_i\mid Z_i=z,Y_i=0]$, $d=m_1-m_0$ and $s_z=\operatorname{Var}[V_i\mid Z_i=z,Y_i=0]$. For independent observations with $Z_i$ independent of $Y_i$, the delta method gives the asymptotic variance

By Assumption 1, I have $q=\mu p$, and Assumptions 1–3 imply that $d=\mathrm{E}[X_i\mid Y_i=0]=(\mu-\mu p)/(1-\mu p)$, so that $1-d=(1-\mu)/(1-\mu p)$. Substituting these values, I obtain (3). Note that the formula can fail without Assumption 1. Let $X_i$, $Y_i$, $W_i$ and $Z_i$ be independent Bernoulli variables with mean $1/2$, and set $V_i=W_i+X_iZ_i$. Then Assumptions 2–4 hold and Assumption 1 does not. In this example $\mu=p=\gamma=1/2$, $s_1=1/2$ and $s_0=1/4$, the asymptotic variance of $\hat\mu$ is $13/16$, and the right side of (3) is $29/24$. The proof of the proposition in Appendix B is the subject of P019-E003.

Intended meaning

The proof of Proposition 1 begins with the words “The proof proceeds by working with linearized variances”, the later lines of (9) carry $+O(n^{-2})$, and the proposition states a limit. The argument of Appendix B is therefore meant to leading order in $n$. The subgroup means are ratios, which are defined only when both subgroups of subjects with $Y_i=0$ are non-empty, and the argument concerns such samples. On this reading it is correct, although its steps are worded as exact statements for a finite sample.

Let $N_{z,0}$ be the number of subjects with $Z_i=z$ and $Y_i=0$, and let $D=\{N_{1,0}>0,\ N_{0,0}>0\}$. I write $m_z$ and $s_z$ for the mean and the variance of $V_i$ given $Z_i=z$ and $Y_i=0$. Under independent sampling, the complement of $D$ has positive probability for every $n$. The sums in (6) and (7) leave out the case of an empty subgroup, and they equal $1-(1-\gamma)^{k}$ and $1-\gamma^{k}$, not 1. Given all the direct answers and assignments, on $D$ the mean $\bar V_{z,0}$ has conditional expectation $m_z$ and conditional variance $s_z/N_{z,0}$. Since $1-\bar Y$ is determined by the direct answers, this implies that $\bar V_{1,0}$ and $\bar V_{0,0}$ are uncorrelated with $1-\bar Y$ conditionally on $D$. This is the lemma as intended.

The first equality of (9) is the formula for the variance of a product of two independent variables, and Lemma 2 does not imply it as an exact identity. Let $h=s_1/N_{1,0}+s_0/N_{0,0}$, which on $D$ is the conditional variance of $\bar V_{1,0}-\bar V_{0,0}$ given all the direct answers and assignments. By the law of total variance, I find that $\operatorname{Var}[\hat\mu\mid D]$ minus the right-hand side of the first equality, with its moments taken conditionally on $D$, equals

This term has the order of the remainder in the later lines of (9), and the limit in (3) does not depend on it. The sentence before (9) cites Lemma 2 for the statement that the two subgroup means are uncorrelated. The lemma says that each is uncorrelated with $1-\bar Y$, and the words “with $1-\bar Y$” are to be supplied.

Since $\gamma=\Pr(Z_i=1)$ is a population quantity, $n\gamma$ is the expected number of treated subjects, which is what the calculation to leading order uses. The realized number is $\sum_iZ_i=n\hat\gamma$, with $\hat\gamma$ as defined on p. 49. In the first list experiment of Study A, 245 of the 500 subjects were treated, while $n\gamma=250$.

Formula (3), the asymptotic normality of $\hat\mu$, Corollary 1 and Proposition 2 are unaffected. Formula (3) also follows from the influence function of $\hat\mu$, without Lemma 2. In the original main analysis every subgroup defined by $(Z_i,Y_i)$ has at least 21 subjects. The hypothesis of Proposition 1 is the subject of P019-E002.