Corrections to: A Note on Dropping Experimental Subjects who Fail a Manipulation Check

The problem

Appendix E announces the unweighted covariate profiles of the subjects, by treatment and by performance on the manipulation check, as the counterpart of the weighted profiles of Appendix D. Table 5 does not contain them. Its 150 numerical entries and its three numbers of observations are those of Table 2, which gives the weighted profiles of the subjects who failed. The cause is one line of the replication code. The script computes the unweighted table failed.tab in lines 1246–1258, and in line 1261, where it formats the table, it overwrites it with the formatted weighted table w.failed.tab. The two files tab_2.txt and tab_5.txt saved with the replication files are identical byte for byte.

The unweighted profiles of the subjects who failed are not printed anywhere in the paper, and they differ from the printed values in 74 of the 75 means. The share of women among these subjects is 42.9, 52.2 and 50.0 percent in the three treatments, where Table 5 prints 24.9, 68.8 and 63.7 percent. Their mean age is 28.6, 35.9 and 30.8 years, where Table 5 prints 41.0, 52.7 and 39.0. Table 5 therefore gives the wrong unweighted composition of the subjects who would be dropped.

Evidence an exact counterexample

The evidence is a recalculation from the public data in exact arithmetic, with two checks of the printed tables against one another. Applying to the public data the recoding of the replication script, with its replacement of missing covariates by the mean over the replication sample, I obtain the 63 means of the 21 covariates that need no confidential input. Among the subjects who failed, $15$ of $35$, $95$ of $182$ and $20$ of $40$ are women. The printed shares cannot be unweighted shares. A count $k$ with $k/35$, $k/182$ or $k/40$ equal to 0.249, 0.688 or 0.637 after rounding would have to lie in $[8.6975,8.7325]$, $[125.125,125.307]$ or $[25.46,25.50]$, and none of these intervals contains an integer.

The twelve means of the age quartiles depend on quartiles of the sample of Press, Sagan and Valentino, which is not public. They can be recovered from Tables 6 and 7. Since every sample there has fewer than 1,000 subjects, each share printed to three decimals fixes one integer count. The count among the subjects who failed is the count among all subjects minus the count among those who passed. The counts are $(30,4,0,1)$, $(122,29,18,13)$ and $(34,2,3,1)$ for the three treatments. Classifying the ages in the public data at the boundaries 39, 50 and 59 reproduces every one of these counts.

For any covariate, $N_a\bar x_a=N_p\bar x_p+N_f\bar x_f$, where the subscripts denote all subjects, those who passed and those who failed. With the printed means of age in Treatment 2, the right side divided by $N_a$ is $(356\times34.963+182\times52.658)/538=40.949$, where Table 7 prints $35.277$. The gap is $5.67$, and rounding allows at most $0.001$. With the recomputed means, all 75 identities hold exactly, and all 150 means of Tables 6 and 7 are reproduced to the printed precision. The one mean of Table 5 that is unchanged is the zero for the third age quartile in Treatment 1.

The standard errors of the paper are bootstrap estimates from 10,000 samples. The standard deviation of the mean of a bootstrap sample has the closed form $\bigl(\sum_i(x_i-\bar x)^2\bigr)^{1/2}/n$. For the share of women in Treatment 1 it is $\bigl((3/7)(4/7)/35\bigr)^{1/2}=0.084$, where Table 5 prints $0.114$.

What survives

Table 2 and the comparison in Section 3 that cites it, with 63.7 and 24.9 percent of women among the subjects who failed under Treatments 3 and 1, are explicitly weighted and are not affected by this error. I could not recompute the weighted values, since the weights require data that are not public. Tables 6 and 7 are produced from other objects of the code, and their means are reproduced. No figure, estimate, bound or theoretical result of the paper uses Table 5.

The problem

Panels B and C show the replication’s weighted estimates, which Section 3 presents as its primary results. The bootstrap in the replication code fills 24 columns. Columns 1 to 12 hold unweighted means, 1 to 6 for the subjects who passed the check and 7 to 12 for all subjects. Columns 13 to 24 hold the weighted means in the same order. The calls that draw Panels B and C centre each bar at a weighted estimate and take its standard error from columns 1 to 6 or 7 to 12. The bars show the uncertainty of the unweighted means. The weighted columns are computed and never drawn. In the published figure the twelve bars have the same widths as the corresponding bars of the unweighted Figure 2, to within 0.01 percentage points. A weighted mean and an unweighted mean of the same observations have different variances in general. Here the weighted mean is the more variable. The weighted standard errors that the authors' own output reports for the same twelve estimates are 1.87 to 2.39 times the standard errors behind the published bars. Someone who judges the precision of the weighted comparison from Figure 1 sees bars about half as wide as the caption describes. In Panel B, for example, the published bars for preference under 90–70 and 90–45 do not overlap (48.3 to 58.7 and 58.9 to 67.3), and the bars from the weighted standard errors do (42.9 to 64.1 and 53.1 to 73.1). Whether two bars overlap is not a test of the difference, and I draw no conclusion about that difference here.

Evidence an exact counterexample

The evidence has four parts. First, I read the code that fills the 24 bootstrap columns and the indices in the plotting calls, and I ran those calls to confirm which columns they select. Second, I measured the published figure: the twelve bars in Panels B and C of Figure 1 are as wide as those of Figure 2, to within 0.0096 percentage points. The third part is an exact example in which the two standard errors differ. For two observations with outcomes 0 and 1 and weights 1 and 9, the bootstrap variance of the unweighted mean is $1/8$ and that of the weighted mean is $33/200$. The fourth part is the output files tab_3.txt and tab_4.txt in the replication files, which report weighted means and weighted bootstrap standard errors for the two outcomes. The table compares the published bars with the bars these standard errors give.

The bars in the fourth column are approximate. The saved standard errors are rounded to three decimals and come from bootstrap runs separate from the one that produced the figure. The exact bars require a new run with the fitted weights, and the weights depend on the data of Press, Sagan and Valentino (2013), which are not public.

What survives

The weighted point estimates survive, and with them the changes on including the subjects who failed. Under 90–70, approval rises from 60.0% to 66.4% and preference from 53.5% to 61.2%. In the full sample the preference estimates under 90–70 and 90–45 are 61.2% and 61.4%. With the approximate bars, these two preference estimates for the full sample remain above 50% at their lower ends. Panel A, the separately bootstrapped bounds of Panels D and E, the unweighted Figure 2 and the results of Section 2 are unaffected by this indexing error. The weighted comparison in Section 3, p. 578, is less precise than the figure shows. The finding does not show that a conclusion of that section reverses.

The problem

In each simulated data set, the function returnbounds_sims() sorts the outcomes of the subjects who pass in each arm and attaches to them the values of their empirical distribution function. It uses these values to select the tails to be trimmed. Let $m_0$ and $m_1$ be the numbers of subjects who pass in the code’s arms 0 and 1, which are the paper’s treatments 1 and 2. For arm 1, the function builds a vector of $m_0$ entries equal to $1/m_1$, where $m_1$ entries are needed. It uses the cumulative sums $1/m_1,2/m_1,\ldots,m_0/m_1$ to select among the $m_1$ outcomes. Suppose that $m_0>m_1$ and the fraction to trim is less than one. Then the entries after the $m_1$-th exceed one, and R selects an outcome that does not exist. The upper endpoint and the width for that data set are missing. Line 182 averages the widths with na.rm = TRUE. The reported number is then an average over the remaining data sets only, and not over all simulations as the caption says. Suppose instead that $m_0<m_1$. Then the selection vector is too short, and R recycles it. This can select outcomes outside the intended tails: the largest outcomes of the arm can be left out of its upper-tail mean and enter its lower-tail mean.

All twelve entries of the column are affected. In rows 5 to 8, the passing rates are 0.6 and 0.8, and the ratio $m_0/m_1$ is typically close to $3/4$ in large samples. The vector then covers only the lowest three quarters of the outcomes of arm 1. The printed widths 1.613, 3.403, 15.247 and 1.569 are close to the large-sample limit of this computation, which is 1.614, 3.427, 15.247 and 1.568. They are 43 to 67 percent below the population widths of the bounds of Proposition 1, which are 2.915, 10.297, 26.889 and 2.859. In the other eight rows the passing rates are equal. About half of the data sets have $m_0>m_1$ and are dropped; among the rest, those with $m_0<m_1$ use a recycled vector. The printed widths are 1.9 to 3.0 percent below the population widths.

As printed, the widths in rows 5 to 8 are smaller than the widths in rows 1 to 4, although the failure rate is higher. This contradicts the statements of Section 2.2 and Appendix C that the width of the bounds increases with the failure rates, for which Appendix C cites rows 1–4, 5–8 and 9–12. The population widths increase from each of these blocks to the next. The statements hold in these designs, but the printed column does not support them.

Evidence an exact counterexample

The evidence consists of two exact examples evaluated with the function as deposited, a derivation of the population widths, and a new simulation in which every data set is passed through the function with and without the repair.

In the first example, the outcomes of the subjects who pass are $(1,2,3,4)$ in arm 0 and $(10,11,12)$ in arm 1, and one subject fails in each arm. The fraction to trim in arm 1 is $(1/5)/(3/4)=4/15$. The function builds the vector $(1/3,2/3,1,4/3)$. All four entries exceed $4/15$, the upper-tail selection is $(10,11,12,\texttt{NA})$, and its mean is NA. With a vector of length three, the selection is $(10,11,12)$ and the mean is $11$. In the second example, the outcomes are $(1,2,3)$ and $(10,11,12,13)$, again with one failure in each arm, and the fraction to trim in arm 1 is $(1/4)/(4/5)=5/16$. The vector $(1/4,1/2,3/4)$ is recycled over four outcomes. The lower-tail selection becomes $(10,11,13)$, with mean $34/3$, and the upper-tail selection $(11,12)$, with mean $23/2$. After the length repair, the selections are $(10,11)$ and $(11,12,13)$, with means $21/2$ and $12$. The spread of arm 1 is $1/6$ in place of $3/2$, and the function’s width for the difference in means is $2/3$ in place of $2$. These comparisons retain the function’s whole-observation cutoff convention.

For the population widths, let $p_0$ and $p_1$ be the two passing rates, and let

be the probability of passing under both treatments. Among the subjects who pass, the outcomes follow the mixtures

The bounds retain the fraction $r_j=(p_0+p_1-1)/p_j$ of $F_j$. Let $Q_j$ be the quantile function of $F_j$. The width of the bounds on the difference in means is

Note that these widths are large-sample values. They are not the averages that a corrected run at a finite sample size would give, and the data sets of the original run are not available.

In the new simulation, I drew 100 data sets with $n=100{,}000$ for each row from the exact joint distribution of $(S(1),S(2))$. I passed each through the function as deposited and through the function with line 138 repaired. The function as deposited returned a width for 39 to 53 of the 100 data sets in the rows with equal passing rates and for all 100 in rows 5 to 8. Its averages lie close to the printed entries. With the repair, every data set gave a finite width, and the averages lie close to the population widths. Among data sets with widths from both functions, the average increase is between 8.9 and 713 Monte Carlo standard errors in the twelve rows.

What survives

The Bias column of Table 1 is computed in other lines of the code and is not affected by this error. The bounds of the application in Section 3 are computed by a different function, returnbounds(), which builds the distribution of each arm from the weights of that arm. The theoretical results and the paper’s warning about dropping subjects do not use the simulation function. The population widths agree with what the paper says about the width in these designs: it grows with the failure rates, and with $\sigma$ and $\lambda$ in the comparisons that the paper draws. All twelve entries of the column have to be recomputed. The repair needs no new assumption.

The problem

Panels D and E of Figures 1 and 2 show, for each condition, the estimated lower and upper bounds on the mean outcome among subjects who would pass the manipulation check under all three treatments. The sentence says that these intervals overlap across all conditions for both outcomes. For approval they do. For preference they do not. In the weighted Figure 1 the interval for 90–90 is about $[0, 19.7]$ percent, against $[41.5, 66.8]$ for 90–70 and $[26.5, 100]$ for 90–45. In the unweighted Figure 2 the three intervals are $[0, 21.4]$, $[31.0, 57.4]$ and $[40.3, 100]$. In both figures the upper bound for 90–90 is below the lower bounds for the other two conditions. The claim then fails whether “all” is read as a common intersection or as pairwise overlap. The estimated bounds separate the condition of equal effectiveness from the two conditions with a nuclear advantage, and only the intervals for 90–70 and 90–45 overlap each other. The two sentences that follow, that the bounds are “largely uninformative” and that “very little is revealed”, are too strong for the preference outcome. They are judgments without a fixed threshold, and I do not say that they are false: the bounds are wide, and they do not order 90–70 against 90–45.

Evidence a recalculation from the source

The evidence is a recalculation from the replication data and a check of the published figures. For Figure 2, recomputing the bounds in exact arithmetic, with the trimming rule of the replication code, I find the preference intervals $[0, 61/285]$, $[87/281, 27/47]$ and $[56/139, 1]$. They agree with the output of the replication code’s own function and with the plotted bars. The lower bounds on the two contrasts with 90–90 are

that is, 9.56 and 18.88 percentage points. With exact fractional trimming at the cut-off (see P010-E017) the two gaps are 9.50 and 18.91 points. For Figure 1 the weighted estimates cannot be recomputed, because the PSV data are withheld from the replication files. The heights of its bars were read from the drawing instructions of the published figure, and the corresponding gaps are about 21.8 and 6.8 points.

What survives

The approval bounds overlap across all three conditions, the preference bounds for 90–70 and 90–45 overlap each other, and all the bounds are wide. The statement that the treatments cannot be fully differentiated stands. The general warning about dropping subjects, the identification results and the comparison of Panels B and C do not depend on this sentence. What changes is the claim that the bounds reveal very little: for preference, the estimated bounds place 90–90 below both other conditions. This is a statement about the estimated bounds only, and it does not settle the question of sampling uncertainty: in Figure 1 the 95% bars on the upper bound for 90–90 overlap the bars on the two lower bounds.

The problem

The target is a mean over a subset of the subjects who pass in arm $z$, and the quantile function $Q_{Y\mid Z=z,S=1}$ is that of those passers. The outer expectations, as printed, condition only on $Z=z$ and on the tail event. They average over every subject of arm $z$ whose outcome lies in the tail, including subjects who failed the check, and such a mean need not bound the target. The sentence after the proposition describes trimming “the empirical distributions of subjects who pass the manipulation check”, and the replication code computes the bounds from passers only. The statement intended has $S=1$ in both conditioning events. The same omission occurs in the final display of the proof on p. 580, where the expectation conditions on the tail event and on $Z=1$ alone.

Evidence an exact counterexample

The evidence is an exact counterexample to the printed display and a derivation of the corrected one.

Counterexample. Let $K=2$, and let $Z$ be independent of the potential outcomes with $\Pr[Z=1]=\Pr[Z=2]=1/2$. Let $X$ be exponential with rate 1, and let $V$ have density $e^{-2|v|}$ on the real line. There are three groups of subjects. With probability $1/4$, $S(1)=S(2)=1$ and $Y(1)=-X$. With probability $1/4$, $S(1)=1$, $S(2)=0$ and $Y(1)=X$. With probability $1/2$, $S(1)=0$, $S(2)=1$ and $Y(1)=V$. In every group $Y(2)$ has density $e^{-|y|}/2$. Then $Y$ is continuous with support the real line and finite moments. The passing rates are $\Pr[S=1\mid Z=1]=1/2$ and $\Pr[S=1\mid Z=2]=3/4$. For $z=1$ the sum in the display, which I call the trimming ratio of the arm, is $(1/4)/(1/2)=1/2<1$. The passers in arm 1 have density $e^{-|y|}/2$, whose median is 0, and both quantiles in the display equal 0. A direct computation gives

The target is $\mathrm{E}[Y(1)\mid S(1)=S(2)=1]=\mathrm{E}[-X]=-1$. The printed lower bound asserts $-3/4\le -1$, which is false. Exchanging $X$ and $-X$ between the first two groups leaves the distribution of $(Y,S,Z)$ unchanged and makes the target equal to 1, while the printed upper bound is $3/4$. The printed interval is $[-3/4,3/4]$ in both cases. With $S=1$ added the interval is $[-1,1]$, and the two examples attain its endpoints.

Derivation. I write $A=\{S(1)=\cdots=S(K)=1\}$, $p_k=\Pr[S=1\mid Z=k]$, $L=1-\sum_{k=1}^K(1-p_k)$ and $Q_z=Q_{Y\mid Z=z,S=1}$. The trimming ratio of arm $z$ equals $1-L/p_z$, and it is below 1 exactly when $L>0$. By the union bound, I have $\Pr[A]\ge L$. Since $A\subseteq\{S(z)=1\}$, the subjects in $A$ are a fraction $r\ge L/p_z$ of the passers in arm $z$. Among all subsets of these passers of fraction $r$, the mean outcome is at least $r^{-1}\int_0^rQ_z(u)\,du$ and at most $r^{-1}\int_{1-r}^1Q_z(u)\,du$. The first average is nondecreasing in $r$ and the second is nonincreasing, and I obtain

For a continuous outcome these two averages are the conditional means of the corrected display. The bounds are sharp. Let $A$ be the lowest, or the highest, fraction $L/p_z$ of the passers in arm $z$, and divide the remaining passers of that arm into disjoint sets of probabilities $1-p_k$, $k\ne z$, on which $S(k)=0$ and the other indicators equal 1. Outside the passers of arm $z$, set $S(z)=0$ and the other indicators equal to 1, and draw the remaining potential outcomes from their observed distributions given passing or failing. This law has the observed distribution of $(Y,S)$ in every arm and attains the endpoint.

What survives

The bounds for a trimming ratio below 1 survive in the form intended, and they are sharp. Adding $S=1$ changes no reported number, since the deposited functions already restrict the bound calculations to passers. This omission does not affect Lemma 1, Corollaries 1 to 5 or the comparisons of point estimates in Section 3. The branch in which the ratio is at least 1 is the subject of P010-E002.

The problem

The variable $Y$ is the observed response of all subjects. Its support can be the whole real line while the outcomes of the subjects who pass in arm $z$ are bounded, because the subjects who fail contribute to the support of $Y$. Every subject who would pass under all conditions passes in arm $z$. The target mean therefore lies between the endpoints of the support of $Y$ given $Z=z$ and $S=1$. When that support is bounded, the target is confined to a finite interval. The printed interval from $-\infty$ to $\infty$ still contains the target, but it is not sharp, and sharpness is what the proposition claims. The error is confined to the branch in which the sum in the display, which I call the trimming ratio of arm $z$, is at least 1.

Evidence an exact counterexample

The evidence is an exact counterexample to the printed branch and a derivation of the corrected one.

Counterexample. Let $K=2$, and let $Z$ be independent of the potential outcomes with $\Pr[Z=1]=\Pr[Z=2]=1/2$. Let $B$ be a Bernoulli variable with $\Pr[B=1]=1/3$, and set $S(1)=S(2)=B$. If $B=1$, let $Y(1)=Y(2)$ have density $b(x)=30x^2(1-x)^2$ on $(0,1)$. If $B=0$, let $Y(1)=Y(2)$ have density $5/[2(1+|v|)^6]$ on the real line. Then the passing rate is $1/3$ in each arm, the trimming ratio is $(2/3)/(1/3)=2\ge 1$, and $\Pr[S(1)=S(2)=1]=1/3>0$. The observed $Y$ has a continuous density that is positive on the whole real line, with $\mathrm{E}[Y]=1/6$, $\mathrm{E}[Y^2]=13/63$ and $\mathrm{E}|Y|^3=19/84$. All the assumptions of the proposition hold, and the printed bounds are $-\infty$ and $\infty$.

Consider any law of the potential outcomes with this observed distribution. The subjects who would pass under both conditions pass in arm $z$, and the passers in arm $z$ have outcomes in $(0,1)$. This implies that

Both endpoints are approached. Fix $t\in(0,1)$ and let $F(t)=10t^3-15t^4+6t^5$ be the distribution function of $b$. Give the pairs $(S(1),S(2))=(1,1)$, $(1,0)$, $(0,1)$ and $(0,0)$ the probabilities $a$, $1/3-a$, $1/3-a$ and $1/3+a$, where $a=F(t)/3$. Let the outcomes in the first group follow $b$ conditional on $(0,t)$, let passing outcomes in the second and third groups follow $b$ conditional on $[t,1)$, and let failing outcomes follow the second density. This law has the same distribution of $(Y,S,Z)$, and the mean in its first group is below $t$. Using $(1-t,1)$ in place of $(0,t)$, I obtain a mean above $1-t$. For $t=1/3$ the two means are $73/306$ and $233/306$. The sharp bounds are therefore 0 and 1, and neither is attained.

Derivation. In general, let the trimming ratio of arm $z$ be at least 1, and let $\ell_z$ and $u_z$ be the essential infimum and supremum of $Y$ given $Z=z$ and $S=1$. Since every subject who would pass under all conditions passes in that arm, I have $\ell_z\le\mathrm{E}[Y(z)\mid S(1)=\cdots=S(K)=1]\le u_z$ whenever the mean is defined. Let $E$ be a bounded set of positive probability under the distribution of the passers in arm $z$. Then there exists a law with the same observed distribution such that the probability of passing under all conditions is positive, and such that the outcomes of the subjects who would pass under all conditions follow that distribution conditional on $E$. Letting $E$ approach either end of the support, I obtain means arbitrarily close to $\ell_z$ and to $u_z$, or beyond any given value when the endpoint is infinite.

What survives

The branch with a trimming ratio below 1 and the estimand are unaffected, and the bounds are sharp in the corrected form. The printed interval from $-\infty$ to $\infty$ remains a valid bound in this branch. The statement that the bounds may have infinite width holds when the outcomes of the passers are unbounded in both directions. No number in the paper comes from this branch. The ratio is below 1 in all 12 simulation designs of Table 1, where its largest value is $2/3$, and in all three arms of the application, weighted and unweighted. The application has binary outcomes.

The problem

The proof bounds $\mathrm{E}[Y(1)\mid U=1,S(1)=1]$ from above, where $U=1$ for the subjects who would also pass under treatments $2,\ldots,K$. The sentence quoted second says that this mean is largest when those subjects occupy the upper tail of $Y(1)$ among the subjects with $S(1)=1$. The displayed bound conditions on the lower tail instead: with $r=\sum_{k=2}^{K}\Pr[S(k)=0]/\Pr[S(1)=1]$, it uses the event $Y(1)\le Q_{Y(1)\mid S(1)=1}(1-r)$, which belongs to the lower bound. The inequality is false as printed. Further, the expectation on the right is not conditioned on $S(1)=1$, and the quantile in the earlier sentence lacks the subscript $Y(1)\mid S(1)=1$. The next display of the proof and the statement of Proposition 1 have the upper tail.

Evidence an exact counterexample

The evidence is an exact counterexample to the printed inequality and a derivation of the corrected one. For the counterexample, let $K=2$ and $S(1)=S(2)=B$, where $B$ is Bernoulli with $\Pr[B=1]=0.8$, and let $Y(1)$ be standard normal and independent of $B$, with randomized assignment. Then $U=B$, the left side is $\mathrm{E}[Y(1)\mid B=1]=0$, and $r=0.2/0.8=0.25$. The right side is the mean of a standard normal variable below its $0.75$ quantile, which is $-\varphi(\Phi^{-1}(0.75))/0.75\approx-0.4237$. The display then asserts $0\le-0.4237$. The corrected right side is $+0.4237$.

For the derivation, I write $A=\{S(1)=1\}$. The event $\{U=0\}\cap A$ is contained in $\bigcup_{k\ge2}\{S(k)=0\}$. By the union bound, I have $\Pr[U=0\mid A]\le\min(1,r)$. If $r<1$, the subjects with $U=1$ then make up a share of at least $1-r$ of $A$. Since $Y(1)$ is continuous given $A$, the mean of $Y(1)$ over a subset of $A$ of conditional probability at least $1-r$ is largest on the upper tail $\{Y(1)\ge Q_{Y(1)\mid A}(r)\}$. This yields the corrected bound, and the lower tail $\{Y(1)\le Q_{Y(1)\mid A}(1-r)\}$ gives the lower bound.

What survives

The union bound, the upper-tail trimming argument and the upper bound stated in Proposition 1 are unaffected; the error is a misprint in one display of the proof. The display that follows has a separate slip in its denominator (P010-E005), and the display of the Proposition itself omits the conditioning on $S=1$ (P010-E001).

The problem

The display restates the preceding one in terms of observed quantities. The ratio in the preceding display is $\sum_{k=2}^{K}\Pr[S(k)=0]/\Pr[S(1)=1]$. By random assignment and SUTVA, I have $\Pr[S(k)=0]=\Pr[S=0\mid Z=k]$ and $\Pr[S(1)=1]=\Pr[S=1\mid Z=1]$. The denominator is therefore $\Pr[S=1\mid Z=1]$ for every $k$, in agreement with the statement of Proposition 1, where it is $\Pr[S=1\mid Z=z]$. As printed, the ratio is the sum of the failure odds of the other arms. This changes the fraction that is trimmed and the condition under which the bound is finite, and the printed expression need not be an upper bound. The conditional mean on the right also lacks the event $S=1$. That omission is the subject of P010-E001.

Evidence a derivation

The evidence is a derivation and two examples with $K=2$. Let $p_k=\Pr[S=1\mid Z=k]$, $A=\{S(1)=1\}$ and $d=\sum_{k=2}^{K}\Pr[S(k)=0]$. Since $\Pr[A\cap\{U=0\}]\le\min\{p_1,d\}$, I have $p_{U^0}\le\min\{1,d/p_1\}$. Suppose that $r_1=d/p_1<1$. Then at least the fraction $1-r_1$ of the subjects in $A$ have $U=1$. For a continuous outcome, the largest mean over a subset of $A$ of relative mass at least $1-r_1$ is the mean of $Y(1)$ above its $r_1$ quantile in $A$. Random assignment and SUTVA imply that $d=\sum_{k=2}^{K}(1-p_k)$, and hence $r_1=\sum_{k=2}^{K}(1-p_k)/p_1$.

In the first example, $p_1=3/5$ and $p_2=4/5$. The correct ratio is $(1/5)/(3/5)=1/3$, and the printed ratio is $(1/5)/(4/5)=1/4$. Let $Y(1)$ be standard normal among the subjects with $S(1)=1$, let $S(2)=1$ for exactly the upper two thirds of these outcomes, and let $S(2)=1$ for every subject with $S(1)=0$. Then $\Pr[S(2)=1]=2/5+2/5=4/5$, and the mean to be bounded is $\mathrm{E}[Y(1)\mid Y(1)\ge\Phi^{-1}(1/3)]=0.5454$. The printed formula, with $S=1$ added to the conditioning event, gives $\mathrm{E}[Y\mid Y\ge\Phi^{-1}(1/4)]=0.4237$, which is below this mean. In the second example, $p_1=9/10$ and $p_2=1/2$. The correct ratio is $5/9$, and the bound is finite. The printed ratio is $1$, and the printed display declares the bound infinite.