Corrections to: On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry

The problem

Reward Functions 1 and 2 are averages over the treated units in a sample of $n$ units, and each is defined to be $0$ when no unit is treated. Section 2 and Appendix B.1 define the agent’s best response as the policy that maximizes the expected reward $E[w(\mathbf{X},\mathbf{T}^\pi,\mathbf{Y})]$, with no conditioning. The first equality in the proof of Proposition 6, and the first in the proof of Proposition 7, replace this expectation by the mean among the treated units. That mean is the expected reward given that at least one unit is treated. The expected reward has the further factor $1-(1-q)^n$, the probability that at least one unit is treated, where $q=E[\pi(X)]$ is the probability that a given unit is treated. This factor depends on the policy and increases with $q$: a policy that treats more units is left less often with the reward $0$. The printed rule maximizes the mean among the treated units and does not account for this factor. It therefore need not maximize the expected reward.

Section 3 states the same rule for both rewards and adds that under $w_{\mathrm{ATT}}$ the agent “only treats patients with the single value $x$ with maximum treatment effect”. The proof of Proposition 2 substitutes the printed rule for the best response. The rows $w_{\mathrm{ATO}}$ and $w_{\mathrm{ATT}}$ of Tables 1 and 2 are computed for the printed rule: the replication files treat the units whose fitted $\mu_1$, or fitted $\mu_1-\mu_0$, equals the largest value in the sample. The captions of the tables present these rows as the utility, the regret and the treatment rate of the best response $\pi^w$.

A recalculation from the replication files shows that the printed rule is the best response for one of the four rows, $w_{\mathrm{ATO}}$ on the Horse Colic data, where 49 cases share the largest fitted $\mu_1$. For the other three rows the best response treats more units than the printed rule. I call the largest welfare of any policy on the fitted data the achievable welfare; it is the utility in the row $w_{\mathrm{TT}}$. For $w_{\mathrm{ATT}}$ on the Horse Colic data, the best response treats nine cases where the printed rule treats one, and it keeps 48.03% of the achievable welfare where the printed rule keeps 5.34%. The printed row therefore understates the welfare under this reward on these data by a factor of about nine. On the Stroke Trial data the best responses keep 1.45% and 2.50% of the achievable welfare, and the printed rows still show correctly that welfare is low under both rewards.

The printed rule does maximize a different objective, the mean among the treated units in the population, when the maximum of $\mu_1$ or $\tau$ is positive and is attained with positive probability. For a fixed policy with $q>0$, the expected reward tends to that mean as $n\to\infty$. The paper states neither qualification.

Evidence an exact counterexample

The evidence is a derivation of the expected reward, an exact counterexample, and a recalculation of the two rows of the tables from the replication files.

Let the units be independent and identically distributed with integrable potential outcomes, and let the treatment indicators be drawn independently given the covariates. For $w_{\mathrm{ATT}}$, suppose that $\hat\mu_0(X_i)$ is integrable and $E[\hat\mu_0(X_i)\mid\mathbf{X},\mathbf{T}^\pi]=\mu_0(X_i)$. I write $s=\mu_1$ for $w_{\mathrm{ATO}}$ and $s=\tau$ for $w_{\mathrm{ATT}}$, and set $q=E[\pi(X)]$, $A=E[s(X)\pi(X)]$ and $N=\sum_{i=1}^nT_i^\pi$. Each unit is treated with probability $q$. By independence, I have $P(N=0)=(1-q)^n$. Let $t\in\{0,1\}^n$ be a treatment vector of positive probability such that at least one unit is treated. Given $\mathbf{T}^\pi=t$, the term of each treated unit in the numerator of the reward ($Y_i$, or $Y_i-\hat\mu_0(X_i)$) has mean $A/q$. The reward is the average of these terms, so its conditional mean is also $A/q$. The reward is $0$ when $N=0$. Summing over $t$, I obtain

and $E[w(\mathbf{X},\mathbf{T}^\pi,\mathbf{Y})]=0$ for $q=0$. Note that the first display in each printed proof keeps only $A/q$.

For the counterexample, I take $n=2$ and two equally likely types $x\in\{0,1\}$, with $Y(0)=0$ and $\hat\mu_0=0$, and with $Y(1)=9/10$ for type $0$ and $Y(1)=1$ for type $1$. Then $\mu_1=\tau$, the two rewards are equal, and $\mu_1$ has a unique maximum, which is positive, at $x=1$. The printed rule treats type $1$ only.

The policy that treats everyone has expected reward $19/20$, which exceeds the expected reward $3/4$ of the printed rule by $1/5$. For the printed rule, the right side of the first display of the printed proofs equals $1$, and the left side equals $3/4$.

Neither of these two policies is the best response. With $u=\pi(0)$ and $v=\pi(1)$, the formula above gives $E[w]=(9u+10v)(4-u-v)/40$, and

for $0\le u,v\le1$, with equality only at $u=17/18$ and $v=1$. The best response treats type $0$ with probability $17/18$ and type $1$ with probability $1$, and it has expected reward $1369/1440$.

On a finite support, the best response is found as follows. In the notation of the corrected statement below, let $F(q)=h_n(q)A_{\max}(q)$ be the largest expected reward among the policies with $E[\pi(X)]=q$, and let $g(q)=1-(1-q)^n$, so that $F(q)=g(q)A_{\max}(q)/q$ for $q>0$. On the segment $Q_{j-1}\le q\le Q_j$, I have $A_{\max}(q)=a_j+s_jq$ with $a_j=\sum_{i<j}p_i(s_i-s_j)\ge0$, and a direct computation gives

for $0<q<1$. The function $H_n$ is nonnegative and nondecreasing, and at $q=Q_j$ the slope of $A_{\max}$ falls from $s_j$ to $s_{j+1}$. These two facts imply that $F'(q)/g'(q)$ is nonincreasing on $(0,1)$. A value of $q$ at which $F'$ changes sign from positive to negative then maximizes $F$ over $[0,1]$. Let $M=s_1>0$ and $p=P(s(X)=M)$, and, if $p<1$, let $s'$ be the largest value of $s$ below $M$. The printed rule is the policy of this form with $q=p$. On $0<q\le p$, I have $F(q)=Mg(q)$, which increases with $q$, and for $p<1$ the right derivative of $F$ at $p$ equals

For $p<1$, the printed rule is therefore a best response if and only if $s'\le M(1-r_n(p))$. If $p=1$, treating everyone is a best response.

In the replication files, both notebooks compute the row $w_{\mathrm{ATO}}$ from the indicator that the fitted $\mu_1$ of a unit equals the largest fitted $\mu_1$ in the sample, and the row $w_{\mathrm{ATT}}$ from the indicator that its fitted $\mu_1-\mu_0$ equals the largest such difference. For the recalculation, I computed the best responses to the rewards as defined on the same fitted values. The population is the empirical distribution of the cases that the notebook analyzes, with the fitted $\mu_0$ and $\mu_1$ of the notebook. There are 255 cases for the Horse Colic data and 7264 patients for the Stroke Trial data. The $n$ units are independent draws from this distribution, with $n$ equal to the number of cases, and $\Pi$ is the set of all functions from the covariate values to $[0,1]$. For each data set and each reward, I located the change in sign of $F'$ and checked the signs on either side in exact rational arithmetic.

On the Horse Colic data, 49 cases have a fitted $\mu_1$ equal to $1$ in the double-precision arithmetic of the notebook. The printed rule treats these 49 cases, and the best response to $w_{\mathrm{ATO}}$ treats the same cases. The nine largest values of the fitted $\mu_1-\mu_0$ lie between $1.9994$ and $2.0000$, and the tenth is $1.9062$. The printed rule treats the one case with the largest value, and the best response to $w_{\mathrm{ATT}}$ treats the nine. The welfare of this best response is $0.0706$, its regret is $0.0764$, and it treats 3.53% of the cases. For $n=255$, the expected reward is $1.2657$ under the printed rule and $1.9996$ under the best response. On the Stroke Trial data, the best response to $w_{\mathrm{ATO}}$ treats four patients with probability $1$ and a fifth with probability $0.8452$, and the best response to $w_{\mathrm{ATT}}$ treats five patients. The printed rule treats one patient under each reward. The condition $s'\le M(1-r_n(p))$ holds for $w_{\mathrm{ATO}}$ on the Horse Colic data and fails for the other three rows.

The paper gives $n=300$ for the Horse Colic data, and the notebook analyzes the 255 cases that remain after its exclusions. With $n=300$ and the same 255 cases, I find that the best responses, the welfare, the regret and the treatment rate are the same.

The notebook for the Stroke Trial data fits a linear model by least squares. I fitted it again from the data, and the coefficients agree with those printed in the stored output of the notebook to within $5\times10^{-10}$. For the Horse Colic data I did not fit the logistic model again. I reconstructed the fitted values from the coefficients, which the stored output of the notebook gives to eight decimal places. The intercept is not printed, and I recovered it from the group means in the stored output. Bounds for the rounding of these numbers were carried through the calculation. For every set of fitted values within the bounds, the best response to $w_{\mathrm{ATT}}$ treats the same nine cases, and it keeps between 48.0296% and 48.0297% of the achievable welfare. The tie among the 49 cases is an effect of rounding in double precision. With the logistic function evaluated to 90 decimal places, the best response to $w_{\mathrm{ATO}}$ for $n=255$ treats 41 cases, and its welfare is still $0$ to fifteen decimal places. The best response to $w_{\mathrm{ATT}}$ does not change.

What survives

The definitions of the two rewards stand, and so does the bound of Proposition 2, $R(\pi^{w_{\mathrm{ATT}}})\le\max_{\pi\in\Pi}V(\pi)$, with the proof given below. The conclusion of Proposition 1 also stands. With $P(X=1)=p\in(0,1)$, $\mu_1(0)=0$, $\mu_1(1)=1$ and $n\ge2$, the unique best response to $w_{\mathrm{ATO}}$ as defined treats type $1$ only, and with $\mu_0(0)=0$ and $\mu_0(1)=B>1$ its regret is $p(B-1)$, which is unbounded in $B$; for $n=1$ every best response has this regret. Under the same conditional unbiasedness condition, Proposition 3 does not depend on the error, because $E[w_{\mathrm{TT}}]=nV(\pi)$ involves no division by the number of treated units. The ranking and information-asymmetry results and their rows in the tables use total-effect rewards and do not depend on it either. These are statements about dependence on this error, not a confirmation of those results.

What changes is the description of the agent under the two average rewards and the reading of the rows $w_{\mathrm{ATO}}$ and $w_{\mathrm{ATT}}$ of Tables 1 and 2. The agent need not treat only the best type. The printed rows summarize the printed rule, subject to the rounding corrections in P006-E018. They are the rows of the best response only for $w_{\mathrm{ATO}}$ on the Horse Colic data. The statement of Section 6 that the utility for $w_{\mathrm{ATO}}$ is close to zero for both datasets stands. On the Stroke Trial data both rewards still lose almost all of the achievable welfare: the best responses keep 1.45% and 2.50% of it and treat about five of the 7264 patients. On the Horse Colic data the best response to $w_{\mathrm{ATT}}$ still has positive regret, but it keeps 48.03% of the achievable welfare, with welfare $0.0706$, regret $0.0764$ and 3.53% of the cases treated. The printed row, with utility $0.00784$ and regret $0.1391$, corresponds to 5.34%.

The problem

Assumption 1 fixes the mean of the random function $\hat\mu_0$ at each fixed argument $x$. The proofs use the mean of $\hat\mu_0$ at the agent’s own treated observations, that is, of $\hat\mu_0(X_i)$ given $T_i^\pi=1$. The first does not imply the second when the estimator depends on the sample on which the agent is scored. In that case $\hat\mu_0(X_i)$ can be biased for $\mu_0(X_i)$ although $\hat\mu_0(x)$ is unbiased for every fixed $x$, so the three highlighted steps need not hold. If Assumption 1 is the only condition on the estimator, then the conclusions drawn from these steps fail as well: the best responses of Propositions 7 and 9, the bound of Proposition 2, the zero regret of Proposition 3 and Theorem 1, the ranking property of Theorem 2, and the bound $2\gamma_{\mathrm{marg}}$ of Theorem 3.

The paper does not exclude such estimators. The auxiliary data on PDF p. 4 are introduced as “a concrete example”, and Appendix A.1, item 2, proposes to add the agent’s own untreated units to the estimate of $\mu_0$. Assumption 1 is a legitimate assumption. What fails is the step from it to the conditional statements.

The examples below do not require an agent who observes the realized estimate before choosing a policy. In each of them the policy is a fixed function of the covariates, as in Section 2, and the expected reward is taken over the estimator as well.

Evidence an exact counterexample

I use exact counterexamples to the printed implication, computed in rational arithmetic, and derive the total-effect identities and regret bound under the corrected assumption.

Example A. Let $n=1$, let $X_1$ take the values 0 and 1 with probability $1/2$ each, and let $Y_1(0)=0$ and $Y_1(1)=1/2$. Then $\mu_0(x)=0$ and $\tau(x)=1/2$ for both values of $x$. Define the estimator by $\hat\mu_0(x)=2\cdot\mathbf{1}\{x=X_1\}-1$. Note that it uses only the covariate, which the principal observes, and that it does not depend on the policy.

For each fixed $x$, I have $E[\hat\mu_0(x)]=(1/2)(1)+(1/2)(-1)=0=\mu_0(x)$, and Assumption 1 holds. However, $\hat\mu_0(X_1)=1$ in both states, so under the treat-all policy $E[\hat\mu_0(X_1)\mid X_1,T_1^\pi=1]=1$, not $\mu_0(X_1)=0$. For a policy with $\pi(0)=p_0$ and $\pi(1)=p_1$, a direct computation gives

The reward is maximized only at $p_0=p_1=0$, and welfare only at $p_0=p_1=1$. The best response has regret $1/2$, which contradicts Propositions 3 and 9, and Theorem 1 with $g\equiv1$. Since $n=1$, the rewards $w_{\mathrm{ATT}}$ and $w_{\mathrm{TT}}$ coincide, and the best response stated in Proposition 7 also fails. If $U_1$ is constant, then $\gamma_{\mathrm{marg}}=0$, and the bound of Theorem 3 would read $1/2\le0$.

Example B. In the setting of Example A, I change two signs: $Y_1(1)=-1/2$ and $\hat\mu_0(x)=1-2\cdot\mathbf{1}\{x=X_1\}$. The estimator is again unbiased at each fixed $x$, and it equals $-1$ at $X_1$. Then $E[w_{\mathrm{TT}}]=E[w_{\mathrm{ATT}}]=(p_0+p_1)/4$ and $V(\pi)=-(p_0+p_1)/4$. The best response treats everyone, and its regret $1/2$ exceeds $\max_{\pi}V(\pi)=0$, which contradicts Proposition 2.

Example C. Let $j$ and $k$ be two agents with one unit each. Their covariates $X_j$ and $X_k$ are independent and take the values 0 and 1 with probability $1/2$ each, as does the reference covariate, so that all weights $g$ equal 1. The untreated outcomes are 0, and the treated outcomes are $1/4$ for agent $j$ and $1/8$ for agent $k$. Both agents are scored with the estimator $\hat\mu_0(x)=\mathbf{1}\{x=X_j\}-\mathbf{1}\{x=X_k\}$, which has mean 0 at each fixed $x$. Then $\max_{\pi_j}E[w_j]=0$ and $\max_{\pi_k}E[w_k]=5/8$, although $\tau^j(x)>\tau^k(x)>0$ for every $x$. The ranking of Theorem 2 is reversed.

The corrected assumption. Suppose that $E[\hat\mu_0(X_i)\mid X_i,U_i,T_i^\pi=1]=\mu_0(X_i)$ whenever $\pi(X_i,U_i)>0$. Since $P(T_i^\pi=1\mid X_i,U_i)=\pi(X_i,U_i)$, I have

and ignorability and consistency imply that $E[T_i^\pi Y_i\mid X_i,U_i]=\pi(X_i,U_i)\,\mu_1(X_i,U_i)$. Subtracting, summing over $i$ and taking expectations, I obtain

Without $U_i$ the right side is $E[\pi(X_i)\tau(X_i)]=V(\pi)$. Reward and welfare then have the same maximizers over any feasible set, which is Proposition 3. The following weighted identity uses the per-unit integrability conditions stated below. With a weight $g(X_i)$, I find in the same way that $E[w^g_{\mathrm{TT}}]=nE[g(X_i)\pi(X_i)\tau(X_i)]$, which is the identity used for Theorems 1 and 2. With $U_i$, I write $\widehat V(\pi)=E[w_{\mathrm{TT}}]/n$. Then

Let $\pi^*$ maximize welfare and $\pi^w$ maximize expected reward. Since $\widehat V(\pi^*)\le\widehat V(\pi^w)$, I conclude that

which is the bound of Theorem 3. If $\hat\mu_0$ is unbiased at each fixed $x$ and independent of the agent’s covariates, private information and treatment randomization, then $E[\hat\mu_0(X_i)\mid X_i=x,U_i=u,T_i^\pi=1]=E[\hat\mu_0(x)]=\mu_0(x)$, and the corrected assumption holds. The last column of the second table shows this for Example A.

What survives

Without information asymmetry, the corrected assumption, with integrable treated estimator residuals, gives the identity $E[w_{\mathrm{TT}}]=nV(\pi)$ and the zero regret of Proposition 3 for any feasible set. With the weighted integrability conditions below, it also gives Theorem 1 for unconstrained policies and positive weights, by the derivation above. Under information asymmetry, it gives the bound $2\gamma_{\mathrm{marg}}$ of Theorem 3. The corrected assumption restricts the joint distribution of the estimator and the agent’s sample. An unbiased estimator from independent auxiliary data, as in the example on PDF p. 4 and in Appendix D.2, satisfies it. An estimator that reuses the agent’s sample need not. The rewards for the average and the total treated outcome do not use the estimator. The experiments of Section 6 take the fitted means as given and do not simulate estimation of a separate baseline, so no numerical correction follows from this defect. This correction concerns only the replacement of $\hat\mu_0$ by $\mu_0$. The best responses to the average rewards are corrected in P006-E001.