{ "claims": [ { "claim": "Rewarding the average treated outcome (ATO), the status-quo quality metric, can produce arbitrarily large welfare regret, because it ignores untreated outcomes and rewards an average rather than a total; accounting for the untreated outcome is necessary.", "claim_id": "P006-H1", "related_corrections": [ "P006-E001", "P006-E007", "P006-E003" ], "tied_to": "Proposition 1; Proposition 6 (ATO best response); Theorem 4 (Appendix A.2); Table 1 row $w_{\\mathrm{ATO}}$" }, { "claim": "The paper states that subtracting an unbiased estimate of the untreated outcome and rewarding the average treatment effect on the treated (ATT) bounds regret by the maximum achievable welfare. It attributes the remaining nonzero regret to the agent treating only the covariate value with the largest treatment effect.", "claim_id": "P006-H2", "related_corrections": [ "P006-E001", "P006-E002" ], "tied_to": "Proposition 2; Proposition 7 (ATT best response); Table 1 row $w_{\\mathrm{ATT}}$" }, { "claim": "Rewarding the total treatment effect, $w_{\\mathrm{TT}}=\\sum_i (Y_i-\\hat\\mu_0(X_i))T_i$, makes the best response welfare maximizing, with zero regret for any feasible policy set, given an unbiased estimate of the conditional mean untreated outcome.", "claim_id": "P006-H3", "related_corrections": [ "P006-E002" ], "tied_to": "Proposition 3; Proposition 9; Lemma 1" }, { "claim": "Used directly to rank agents, $w_{\\mathrm{TT}}$ violates rank preservation because population size and case mix affect scores. The weighted reward $w^g_{\\mathrm{TT}}$ retains zero regret for any positive weight $g$, and reweighting to a reference covariate distribution satisfies uniform rank preservation and, for nonnegative treatment effects, relative rank preservation.", "claim_id": "P006-H4", "related_corrections": [ "P006-E005", "P006-E013" ], "tied_to": "Definitions 1-2; Proposition 4; Theorem 1; Theorem 2" }, { "claim": "When the agent observes covariates $U$ that the principal does not, an unbiased estimate of $\\mu_0(x)$ need not give zero regret. Regret under $w_{\\mathrm{TT}}$ is at most $2\\gamma_{\\mathrm{marg}}$, where $\\gamma_{\\mathrm{marg}}$ bounds $E[|\\mu_0(X)-\\mu_0(X,U)|]$; the bound is tight up to a constant, with examples having regret at least $\\gamma_{\\mathrm{marg}}-\\epsilon$.", "claim_id": "P006-H5", "related_corrections": [ "P006-N006", "P006-E002" ], "tied_to": "Assumption 2; Theorem 3; Proposition 5; Lemma 2; Proposition 10" }, { "claim": "If the principal estimates the untreated outcome from the agent's own untreated units, the bounded-heterogeneity condition in Assumption 4 gives regret under $w_{\\mathrm{TT}}$ of at most $2\\gamma_{\\max}$.", "claim_id": "P006-H6", "related_corrections": [ "P006-E006", "P006-E014", "P006-N001" ], "tied_to": "Section 5, 'Information asymmetry and confounding'; Appendix D.3: Assumption 4; Theorem 5; Lemma 3" }, { "claim": "On two clinical datasets with synthetic potential outcomes, Tables 1 and 2 report low welfare for the policies used with $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ relative to $w_{\\mathrm{TT}}$, which has zero reported regret. The reported achievable welfare is higher for Horse Colic than for Stroke Trial.", "claim_id": "P006-H7", "related_corrections": [ "P006-E001", "P006-E008", "P006-E009", "P006-E010", "P006-E011", "P006-E018", "P006-N003" ], "tied_to": "Table 1; Table 2; Appendix E.2; Section 6 text (benefit counts and average benefit)" }, { "claim": "Under information asymmetry, conditioning the counterfactual estimate on a subset of features, such as demographics, can leave regret unchanged or increase it relative to conditioning on nothing (regret amplification); more conditioning need not be beneficial.", "claim_id": "P006-H8", "related_corrections": [ "P006-E011", "P006-N003" ], "tied_to": "Table 1 and Table 2, rows $w_{\\mathrm{TT}}$ (no info) and $w_{\\mathrm{TT}}$ (demographic info); Figure 2; Appendix E.5" }, { "claim": "In the reported experiments, regret falls as features are accumulated in increasing order of their individual $\\gamma_{\\mathrm{marg}}$ and falls substantially before half the features are used. Approximate $\\gamma_{\\mathrm{marg}}$ values may therefore be a useful heuristic for choosing covariates to condition on.", "claim_id": "P006-H9", "related_corrections": [ "P006-N009", "P006-N004", "P006-E015" ], "tied_to": "Figure 1; Figure 3; Appendix E.4; Appendix E.5" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Proposition 6 (ATO Best Response), its displayed rule, and the first display of the proof. One red box marks the two cases of the rule: 1 if x is in the arg max of mu 1 and mu 1 of x is positive, 0 otherwise. A second red box marks the middle expression of the proof's display, 'equals E of E of Y i given T i equals 1 and X i, given T i equals 1', which follows 'E of w ATO'.", "caption": "Appendix B.3, PDF p. 15: Proposition 6 and the first display of its proof. The upper mark is the printed best response to $w_{\\mathrm{ATO}}$. The lower mark is the equality that replaces the expected reward by the mean among the treated units.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001.png" }, { "alt": "Crop of Proposition 7 (ATT Best Response), its displayed rule, and the first two lines of the proof. One red box marks the two cases of the rule: 1 if x is in the arg max of tau and tau of x is positive, 0 otherwise. A second red box marks the right side of the first line of the proof, 'equals E of Y i minus mu hat 0 of X i, given T i equals 1'.", "caption": "Appendix B.3, PDF p. 16: Proposition 7 and the first display of its proof. The upper mark is the printed best response to $w_{\\mathrm{ATT}}$, and the lower mark is the corresponding equality.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-b.png" }, { "alt": "Crop of the left column showing 'Reward Function 1 (ATO)', whose definition has two cases, the average of the treated outcomes when at least one unit is treated and 0 otherwise, followed by the statement of Proposition 1. A red box marks the sentence 'The agent's unconstrained best response is pi of x equals the indicator that x is in the arg max of mu 1 and mu 1 of x is positive.'", "caption": "Section 3, PDF p. 4, left column: Reward Function 1 and the best response stated after it. The second case of the definition sets the reward to $0$ when nobody is treated. The marked sentence gives the rule.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-c.png" }, { "alt": "Crop of the first seven lines of the left column. A red box marks the sentence 'The agent's unconstrained best response is pi of x equals the indicator that x is in the arg max of tau and tau of x is positive.' Below it are the sentence 'The resulting regret is bounded, but still not zero.' and the statement of Proposition 2 (ATT Regret).", "caption": "Section 3, PDF p. 5, left column, top: the best response stated for Reward Function 2 (marked), followed by Proposition 2.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-d.png" }, { "alt": "Crop of six lines of the left column. Two red boxes mark the words 'the agent only treats patients with the single value' in the sentence 'This means that in the best response, the agent only treats patients with the single value x with maximum treatment effect tau of x.'", "caption": "Section 3, PDF p. 5, left column: the sentence that describes the best response to $w_{\\mathrm{ATT}}$ in words.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-e.png" }, { "alt": "Crop of Table 1 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility V of pi w, regret R of pi w, and the realized treatment rate. The table has the columns Utility, Regret and Treat rate for the Horse Colic dataset and for the Stroke Trial dataset, and five rows. A red box surrounds the first two rows: w ATO with 0.00000, 0.1470, 0.1922 and 0.00004, 0.0251, 0.0001; w ATT with 0.00784, 0.1391, 0.0039 and 0.00013, 0.0250, 0.0001. Below the box are the rows w TT, with 0.14695, 0.000, 0.2431 and 0.02518, 0.000, 0.1872, w TT (no info) and w TT (demographic info).", "caption": "Table 1, PDF p. 9, with its caption. The mark surrounds the rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$, which the caption describes as the utility, the regret and the treatment rate of the best response $\\pi^w$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-f.png" }, { "alt": "Line chart. The horizontal axis is the probability of treating the lower type, from 0 to 1, and the vertical axis is the reward, with labelled ticks from 0.70 to 1.00. A solid blue curve, the expected reward as defined, rises from 3/4 at 0 to a maximum of 1369/1440 at 17/18 and ends at 19/20 at 1. A dashed orange curve, the mean among the treated units, falls from 1 at 0 to 19/20 at 1. At 0 an open orange circle is labelled 'printed rule, value in the printed proofs: 1' and a filled blue circle is labelled 'printed rule, expected reward: 3/4'. A blue diamond marks the maximum and a blue square marks 'treat everyone: 19/20'.", "caption": "The counterexample with two units: the reward as a function of the probability $\\pi(0)$ of treating type $0$, when type $1$ is always treated. The solid curve is the expected reward as defined. The dashed curve is the mean among the treated units, which is the expression in the printed proofs. The printed rule is the point $\\pi(0)=0$, where the printed proofs give $1$ and the expected reward is $3/4$; the maximum of the expected reward, $1369/1440$, is at $\\pi(0)=17/18$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E001-reward.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T1", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T1" }, { "id": "P006-T2", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T2" }, { "id": "P006-T3", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T3" }, { "id": "P006-T18", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T18" }, { "id": "P006-T26", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T26" } ], "headline": "Because the rewards $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ are zero when nobody is treated, their expected value is $[1-(1-q)^n]A/q$, where $q=E[\\pi(X)]$, $A=E[s(X)\\pi(X)]$ and $s$ is $\\mu_1$ or $\\tau$; the rule printed as the agent’s best response (treat only the values of $x$ with the largest positive $s(x)$) need not maximize it, and on the Horse Colic data the best response to $w_{\\mathrm{ATT}}$ keeps 48.03% of the achievable welfare, where the printed rule keeps 5.34%.", "id": "P006-E001", "location": "Section 3, best responses stated after Reward Functions 1 and 2, PDF pp. 4–5; Appendix B.3, Propositions 6 and 7 and the first display of each proof, PDF pp. 15–16; proof of Proposition 2, PDF pp. 17–18; rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Table 1 (PDF p. 9) and Table 2 (PDF p. 27)", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Major", "subject": "paper", "target_ids": [ "P006-T1", "P006-T18", "P006-T2", "P006-T26", "P006-T3" ], "text": { "as_printed": "Section 3, PDF p.~4:\n\\[\nw_{\\mathrm{ATO}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})=\\begin{cases}\\dfrac{\\sum_{i=1}^nY_iT_i^\\pi}{\\sum_{i=1}^nT_i^\\pi} & \\sum_{i=1}^nT_i^\\pi>0,\\\\[1ex] 0 & \\text{otherwise.}\\end{cases}\n\\]\n``The agent's unconstrained best response is \\hl{$\\pi^{w_{\\mathrm{ATO}}}(x)=\\mathbb{1}(x\\in\\arg\\max_x\\mu_1(x)\\text{ and }\\mu_1(x)>0)$}.''\n\nSection 3, PDF pp.~4--5:\n\\[\nw_{\\mathrm{ATT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})=\\begin{cases}\\dfrac{\\sum_{i=1}^n(Y_i-\\hat\\mu_0(X_i))T_i^\\pi}{\\sum_{i=1}^nT_i^\\pi} & \\sum_{i=1}^nT_i^\\pi>0,\\\\[1ex] 0 & \\text{otherwise.}\\end{cases}\n\\]\n``The agent's unconstrained best response is \\hl{$\\pi^{w_{\\mathrm{ATT}}}(x)=\\mathbb{1}(x\\in\\arg\\max_x\\tau(x)\\text{ and }\\tau(x)>0)$}. \\ldots\\ This means that in the best response, \\hl{the agent only treats patients with the single value $x$} with maximum treatment effect $\\tau(x)$.''\n\nAppendix B.3, PDF p.~15: \\textbf{Proposition 6} (ATO Best Response). Suppose $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and suppose $X$ is a discrete random variable supported on $\\mathcal{X}$. Then the agent's best response to $w_{\\mathrm{ATO}}$ is:\n\\[\n\\pi^{w_{\\mathrm{ATO}}}(x)=\\hl{\\begin{cases}1 & \\text{if }x\\in\\arg\\max_x\\mu_1(x)\\text{ and }\\mu_1(x)>0\\\\ 0 & \\text{otherwise.}\\end{cases}}\n\\]\n\\emph{Proof.} By the tower property,\n\\[\nE[w_{\\mathrm{ATO}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]\\hl{{}=E[E[Y_i\\mid T_i^\\pi=1,X_i]\\mid T_i^\\pi=1]}=\\int E[Y_i\\mid T_i^\\pi=1,X_i]\\,dP_{X_i\\mid T_i^\\pi=1}.\n\\]\n\nAppendix B.3, PDF p.~16: \\textbf{Proposition 7} (ATT Best Response). Suppose $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and suppose $X$ is a discrete random variable supported on $\\mathcal{X}$. Then the agent's best response to $w_{\\mathrm{ATT}}$ is:\n\\[\n\\pi^{w_{\\mathrm{ATT}}}(x)=\\hl{\\begin{cases}1 & \\text{if }x\\in\\arg\\max_x\\tau(x)\\text{ and }\\tau(x)>0\\\\ 0 & \\text{otherwise.}\\end{cases}}\n\\]\n\\emph{Proof.}\n\\[\nE[w_{\\mathrm{ATT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]\\hl{{}=E[Y_i-\\hat\\mu_0(X_i)\\mid T_i^\\pi=1]}\n\\]\n\nTable 1, PDF p.~9: ``For each reward function $w$, we report utility $V(\\pi^w)$, regret $R(\\pi^w)$, and the realized treatment rate $P(T^{\\pi^w}=1)$.'' The rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ are the same in Table 2, PDF p.~27.\n\\begin{audittable}{lrrrrrr}{Tables 1 and 2, first two rows. The first three columns of numbers are headed ``Horse Colic dataset'' and the last three ``Stroke Trial dataset''.}\nReward function & Utility & Regret & Treat rate & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & 0.00000 & 0.1470 & 0.1922 & \\hl{0.00004} & \\hl{0.0251} & \\hl{0.0001} \\\\\n$w_{\\mathrm{ATT}}$ & \\hl{0.00784} & \\hl{0.1391} & \\hl{0.0039} & \\hl{0.00013} & \\hl{0.0250} & \\hl{0.0001}\n\\end{audittable}", "corrected": "\\textbf{Propositions 6 and 7.} Suppose that $\\hl{n\\ge1}$, the units are independent and identically distributed with \\hl{integrable potential outcomes}, and the treatment indicators are drawn independently given the covariates. For $w_{\\mathrm{ATT}}$, suppose also that, for every admissible policy, $\\hat\\mu_0(X_i)$ is \\hl{integrable} and $E[\\hat\\mu_0(X_i)\\mid\\mathbf{X},\\mathbf{T}^\\pi]=\\mu_0(X_i)$, as holds for a pointwise unbiased estimator computed from independent auxiliary data with policies fixed independently of the estimator (this condition is the subject of \\fid{P006-E002}). Let $s=\\mu_1$ for $w_{\\mathrm{ATO}}$ and $s=\\tau$ for $w_{\\mathrm{ATT}}$, and for a policy $\\pi$ write $q=E[\\pi(X)]$ and $A=E[s(X)\\pi(X)]$. Then\n\\[\nE[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\begin{cases}{}\\hl{\\bigl[1-(1-q)^n\\bigr]}\\,\\dfrac{A}{q} & q>0,\\\\[1ex] 0 & q=0,\\end{cases}\n\\]\nand a best response, when one exists, is \\hl{a policy in $\\Pi$ that maximizes this expression}. It need not treat only the values of $x$ that maximize $s$.\n\nSuppose in addition that $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$ and that $X$ takes finitely many values, with probabilities $p_1,\\dots,p_K>0$ and with the values of $s$ ordered as $s_1\\ge\\dots\\ge s_K$. Set $Q_j=p_1+\\dots+p_j$ and $B_j=p_1s_1+\\dots+p_js_j$, with $Q_0=B_0=0$, and $h_n(q)=\\sum_{k=0}^{n-1}(1-q)^k$, which equals $[1-(1-q)^n]/q$ for $q>0$. Among the policies with $E[\\pi(X)]=q\\in[Q_{j-1},Q_j]$, the largest value of $A$ is $A_{\\max}(q)=B_{j-1}+s_j(q-Q_{j-1})$. It is attained by the policy that treats the first $j-1$ values, treats the $j$th value with probability $(q-Q_{j-1})/p_j$, and treats no other value. A best response is a policy of this form with $q$ chosen to maximize $h_n(q)A_{\\max}(q)$ over $0\\le q\\le1$.\n\nLet $M=s_1$ and $p=P(s(X)=M)$, and suppose that $M>0$. The rule that treats exactly the values of $x$ with $s(x)=M$ is a best response if and only if every value of $x$ with $s(x)0$. The policy $\\pi\\equiv0$ belongs to $\\Pi$ and has expected reward $0$, so a best response $\\pi^{w_{\\mathrm{ATT}}}$ has expected reward at least $0$ and therefore $V(\\pi^{w_{\\mathrm{ATT}}})\\ge0$. Hence $R(\\pi^{w_{\\mathrm{ATT}}})=\\max_{\\pi\\in\\Pi}V(\\pi)-V(\\pi^{w_{\\mathrm{ATT}}})\\le\\max_{\\pi\\in\\Pi}V(\\pi)$.\n\n\\textbf{Tables 1 and 2.} The two rows are computed for the best responses to the rewards as defined, using the fitted values in the double-precision arithmetic of the notebooks, with the $n$ units drawn independently from the empirical distribution of the analyzed cases: $n=255$ for the Horse Colic data, where $n=300$ gives the same row, and $n=7264$ for the Stroke Trial data.\n\\begin{audittable}{lrrrrrr}{Tables 1 and 2, first two rows, for the best responses to the rewards as defined. The first three columns of numbers are for the Horse Colic dataset and the last three for the Stroke Trial dataset.}\nReward function & Utility & Regret & Treat rate & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & 0.00000 & 0.1470 & 0.1922 & \\hl{0.00036} & \\hl{0.0248} & \\hl{0.0007} \\\\\n$w_{\\mathrm{ATT}}$ & \\hl{0.07058} & \\hl{0.0764} & \\hl{0.0353} & \\hl{0.00063} & \\hl{0.0246} & \\hl{0.0007}\n\\end{audittable}\nThe rows as printed report the utility, the regret and the treatment rate of the rule that treats only the units with the largest fitted $\\mu_1$, or the largest fitted $\\mu_1-\\mu_0$. With the corrected rows, the utility for $w_{\\mathrm{ATO}}$ is close to zero for both datasets, as Section 6 states. For $w_{\\mathrm{ATT}}$, the best response keeps 2.50\\% of the achievable welfare on the Stroke Trial data and 48.03\\% on the Horse Colic data, where its regret is $0.0764$ and not $0.1391$.", "evidence": "The evidence is a derivation of the expected reward, an exact counterexample, and a recalculation of the two rows of the tables from the replication files.\n\nLet the units be independent and identically distributed with integrable potential outcomes, and let the treatment indicators be drawn independently given the covariates. For $w_{\\mathrm{ATT}}$, suppose that $\\hat\\mu_0(X_i)$ is integrable and $E[\\hat\\mu_0(X_i)\\mid\\mathbf{X},\\mathbf{T}^\\pi]=\\mu_0(X_i)$. I write $s=\\mu_1$ for $w_{\\mathrm{ATO}}$ and $s=\\tau$ for $w_{\\mathrm{ATT}}$, and set $q=E[\\pi(X)]$, $A=E[s(X)\\pi(X)]$ and $N=\\sum_{i=1}^nT_i^\\pi$. Each unit is treated with probability $q$. By independence, I have $P(N=0)=(1-q)^n$. Let $t\\in\\{0,1\\}^n$ be a treatment vector of positive probability such that at least one unit is treated. Given $\\mathbf{T}^\\pi=t$, the term of each treated unit in the numerator of the reward ($Y_i$, or $Y_i-\\hat\\mu_0(X_i)$) has mean $A/q$. The reward is the average of these terms, so its conditional mean is also $A/q$. The reward is $0$ when $N=0$. Summing over $t$, I obtain\n\\[\nE[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\bigl[1-(1-q)^n\\bigr]\\frac{A}{q}\\qquad\\text{for }q>0,\n\\]\nand $E[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=0$ for $q=0$. Note that the first display in each printed proof keeps only $A/q$.\n\nFor the counterexample, I take $n=2$ and two equally likely types $x\\in\\{0,1\\}$, with $Y(0)=0$ and $\\hat\\mu_0=0$, and with $Y(1)=9/10$ for type $0$ and $Y(1)=1$ for type $1$. Then $\\mu_1=\\tau$, the two rewards are equal, and $\\mu_1$ has a unique maximum, which is positive, at $x=1$. The printed rule treats type $1$ only.\n\\begin{audittable}{llrr}{The counterexample with two units. Each pair of types has probability $1/4$.}\nTypes $(X_1,X_2)$ & Treated under the printed rule & Reward, printed rule & Reward, everyone treated \\\\\n$(0,0)$ & nobody & $0$ & $9/10$ \\\\\n$(0,1)$ & unit 2 & $1$ & $19/20$ \\\\\n$(1,0)$ & unit 1 & $1$ & $19/20$ \\\\\n$(1,1)$ & both units & $1$ & $1$ \\\\\nExpected reward & & $3/4$ & $19/20$\n\\end{audittable}\nThe policy that treats everyone has expected reward $19/20$, which exceeds the expected reward $3/4$ of the printed rule by $1/5$. For the printed rule, the right side of the first display of the printed proofs equals $1$, and the left side equals $3/4$.\n\nNeither of these two policies is the best response. With $u=\\pi(0)$ and $v=\\pi(1)$, the formula above gives $E[w]=(9u+10v)(4-u-v)/40$, and\n\\[\n\\frac{1369}{1440}-\\frac{(9u+10v)(4-u-v)}{40}=\\frac{9}{40}\\Bigl(u-\\frac{17}{18}\\Bigr)^2+\\frac{(1-v)(30-19u-10v)}{40}\\ge0\n\\]\nfor $0\\le u,v\\le1$, with equality only at $u=17/18$ and $v=1$. The best response treats type $0$ with probability $17/18$ and type $1$ with probability $1$, and it has expected reward $1369/1440$.\n\nOn a finite support, the best response is found as follows. In the notation of the corrected statement below, let $F(q)=h_n(q)A_{\\max}(q)$ be the largest expected reward among the policies with $E[\\pi(X)]=q$, and let $g(q)=1-(1-q)^n$, so that $F(q)=g(q)A_{\\max}(q)/q$ for $q>0$. On the segment $Q_{j-1}\\le q\\le Q_j$, I have $A_{\\max}(q)=a_j+s_jq$ with $a_j=\\sum_{i0$ and $p=P(s(X)=M)$, and, if $p<1$, let $s'$ be the largest value of $s$ below $M$. The printed rule is the policy of this form with $q=p$. On $00$, the expected reward tends to that mean as $n\\to\\infty$. The paper states neither qualification.", "what_survives": "The definitions of the two rewards stand, and so does the bound of Proposition 2, $R(\\pi^{w_{\\mathrm{ATT}}})\\le\\max_{\\pi\\in\\Pi}V(\\pi)$, with the proof given below. The conclusion of Proposition 1 also stands. With $P(X=1)=p\\in(0,1)$, $\\mu_1(0)=0$, $\\mu_1(1)=1$ and $n\\ge2$, the unique best response to $w_{\\mathrm{ATO}}$ as defined treats type $1$ only, and with $\\mu_0(0)=0$ and $\\mu_0(1)=B>1$ its regret is $p(B-1)$, which is unbounded in $B$; for $n=1$ every best response has this regret. Under the same conditional unbiasedness condition, Proposition 3 does not depend on the error, because $E[w_{\\mathrm{TT}}]=nV(\\pi)$ involves no division by the number of treated units. The ranking and information-asymmetry results and their rows in the tables use total-effect rewards and do not depend on it either. These are statements about dependence on this error, not a confirmation of those results.\n\nWhat changes is the description of the agent under the two average rewards and the reading of the rows $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ of Tables 1 and 2. The agent need not treat only the best type. The printed rows summarize the printed rule, subject to the rounding corrections in \\fid{P006-E018}. They are the rows of the best response only for $w_{\\mathrm{ATO}}$ on the Horse Colic data. The statement of Section 6 that the utility for $w_{\\mathrm{ATO}}$ is close to zero for both datasets stands. On the Stroke Trial data both rewards still lose almost all of the achievable welfare: the best responses keep 1.45\\% and 2.50\\% of it and treat about five of the 7264 patients. On the Horse Colic data the best response to $w_{\\mathrm{ATT}}$ still has positive regret, but it keeps 48.03\\% of the achievable welfare, with welfare $0.0706$, regret $0.0764$ and 3.53\\% of the cases treated. The printed row, with utility $0.00784$ and regret $0.1391$, corresponds to 5.34\\%." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E001", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines of the right column. Two lines end a paragraph: also have access to an unbiased estimator of the mean conditional untreated potential outcome. Then: Assumption 1. The principal accesses an estimator mu-hat-zero of x which is unbiased: E of mu-hat-zero of x equals mu-zero of x, for all x in the set X. A red outline surrounds this equation. Below, the next paragraph begins: In general, obtaining an unbiased estimator mu-hat-zero of x can be difficult, but circumstances under which causal in-.", "caption": "Assumption 1, PDF p. 4, right column. The outline marks the condition, which is stated for each fixed $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002.png" }, { "alt": "Crop of the start of a proof. A two-line display gives the expected ATT reward as E of Y sub i minus mu-hat-zero of X sub i given T sub i equals 1, and then as E of Y sub i given T sub i equals 1, minus E of the conditional expectation of mu-hat-zero of X sub i given T sub i equals 1 and X sub i, given T sub i equals 1. The next line reads: Since E of mu-hat-zero of x equals mu-zero of x for all x in X, we have E of mu-hat-zero of X sub i given T sub i equals 1 and X sub i, equals mu-zero of X sub i. Thus, followed by a display in which mu-zero has replaced mu-hat-zero. A red outline surrounds the equation that follows the words we have.", "caption": "Proof of Proposition 7, PDF p. 16. The outline marks the conditional statement that is inferred from unbiasedness at each fixed $x$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-b.png" }, { "alt": "Crop of the start of a proof. A display gives the expected TT reward as n times E of Y sub i minus mu-hat-zero of X sub i given T sub i equals 1, times the probability that T sub i equals 1. The text continues: Since we have ignorability in X sub i, we have E of Y sub i given T sub i equals 1 equals E of mu-one of X sub i given T sub i equals 1 (shown in more detail in the above proofs). Since E of mu-hat-zero of x equals mu-zero of x for all x in X, we have E of mu-hat-zero of X sub i given T sub i equals 1, equals E of mu-zero of X sub i given T sub i equals 1. Thus, followed by a display that ends with n V of pi. A red outline surrounds the equation that follows the second we have.", "caption": "Proof of Proposition 9, PDF p. 17. The outline marks the same inference for the total-treatment-effect reward.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-c.png" }, { "alt": "Crop of three displayed lines under the words By the tower property. First line: E of pi of X and U times mu-hat-zero of X equals E of the conditional expectation of mu-hat-zero of X sub i times pi of X sub i and U sub i, given X sub i and U sub i. Second line: equals E of pi of X sub i and U sub i times the conditional expectation of mu-hat-zero of X sub i given X sub i and U sub i. Third line: equals E of pi of X sub i and U sub i times mu-zero of X sub i. A red outline surrounds the third line.", "caption": "Proof of Theorem 3, PDF p. 22. The outline marks the last equality, which replaces $E[\\hat\\mu_0(X_i)\\mid X_i,U_i]$ by $\\mu_0(X_i)$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E002-d.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T4", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T4" }, { "id": "P006-T5", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T5" }, { "id": "P006-T6", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T6" }, { "id": "P006-T7", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T7" }, { "id": "P006-T8", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T8" }, { "id": "P006-T27", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T27" } ], "headline": "Assumption 1, unbiasedness of $\\hat\\mu_0(x)$ at each fixed $x$, does not imply the unbiasedness at the agent’s treated observations that the proofs use; conditional unbiasedness at those observations repairs the estimator substitution in the zero-regret, ranking and $2\\gamma_{\\mathrm{marg}}$ proofs, and is supplied by a pointwise unbiased estimator from independent auxiliary data.", "id": "P006-E002", "location": "Assumption 1, PDF p. 4, right column; proofs of Proposition 7 (PDF p. 16), Proposition 9 (PDF p. 17) and Theorem 3 (PDF p. 22)", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [ { "classification_after": { "class": "error", "severity": "Moderate" }, "classification_before": { "class": "error", "severity": "Moderate" }, "note": "The integrability conditions were revised; the classification is unchanged (error, Moderate). How this came to light: comparing the Lean statement with the audit target exposed a gap between integrability of each weighted reward summand and finiteness of the total expected reward. Two checks confirmed a cancellation counterexample in which the weighted identity and the asserted optimality fail despite finite total expectations. Why it changed: the checked repair explicitly requires integrable treated estimator residuals, integrable $g\\tau$, and integrable individual weighted summands. Integrability of the estimator at untreated evaluations is not required. The best-response and zero-regret conclusions hold under these sufficient conditions. P. M. Aronow delegated this decision to the model session that runs the audit, which adopted the amendment recommended by the two checks; this note was written by that session." } ], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P006-T27", "P006-T4", "P006-T5", "P006-T6", "P006-T7", "P006-T8" ], "text": { "as_printed": "\\emph{Assumption 1 (PDF p.~4).} The principal accesses an estimator $\\hat\\mu_0(x)$ which is unbiased:\n\\[\n\\hl{E[\\hat\\mu_0(x)] = \\mu_0(x)\\ \\ \\forall x\\in\\mathcal X}.\n\\]\n\n\\emph{Proof of Proposition 7 (PDF p.~16).} Since $E[\\hat\\mu_0(x)] = \\mu_0(x)$ for all $x\\in\\mathcal X$, we have\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1,X_i] = \\mu_0(X_i)}.\n\\]\nThus, \\ldots\n\n\\emph{Proof of Proposition 9 (PDF p.~17).} Since $E[\\hat\\mu_0(x)] = \\mu_0(x)$ for all $x\\in\\mathcal X$, we have\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1] = E[\\mu_0(X_i)\\mid T_i^\\pi=1]}.\n\\]\nThus, \\ldots\n\n\\emph{Proof of Theorem 3 (PDF p.~22).} By the tower property,\n\\[\n\\begin{aligned}\nE[\\pi(X,U)\\hat\\mu_0(X)] &= E[E[\\hat\\mu_0(X_i)\\pi(X_i,U_i)\\mid X_i,U_i]] \\\\\n&= E[\\pi(X_i,U_i)E[\\hat\\mu_0(X_i)\\mid X_i,U_i]] \\\\\n&= \\hl{E[\\pi(X_i,U_i)\\mu_0(X_i)]}.\n\\end{aligned}\n\\]", "corrected": "\\emph{Assumption 1.} The principal accesses an estimator $\\hat\\mu_0$ which is unbiased at the agent's treated observations: for every feasible treatment rule $\\pi$ and every unit $i$,\n\\[\n\\hl{E[\\hat\\mu_0(X_i)\\mid X_i,T_i^\\pi=1]=\\mu_0(X_i)\\quad\\text{whenever }\\pi(X_i)>0.}\n\\]\nUnder information asymmetry (Section 5), the condition is $E[\\hat\\mu_0(X_i)\\mid X_i,U_i,T_i^\\pi=1]=\\mu_0(X_i)$ whenever $\\pi(X_i,U_i)>0$. For the baseline substitution in the average reward $w_{\\mathrm{ATT}}$, the condition is imposed given the full vectors: $E[\\hat\\mu_0(X_i)\\mid\\mathbf X,\\mathbf T^\\pi]=\\mu_0(X_i)$ for every treated unit $i$. Here $\\pi$ is a fixed function, as in Section 2, and the expectations are over the sample, the treatment randomization and the estimator; if the estimator uses data generated under $\\pi$, they are taken under the joint distribution that $\\pi$ induces.\n\nIt suffices that $\\hat\\mu_0$ is unbiased at each fixed $x$, that is, $E[\\hat\\mu_0(x)]=\\mu_0(x)$ for all $x\\in\\mathcal X$, and is computed from auxiliary data independent of the agent's covariates, private information and treatment randomization. For an estimator that uses the agent's own sample, such as the one in Appendix A.1, item 2, the condition has to be checked separately.\n\n\\emph{Proofs of Propositions 7 and 9 (PDF pp.~16--17).} \\hl{By Assumption 1}, we have $E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1,X_i]=\\mu_0(X_i)$, and hence $E[\\hat\\mu_0(X_i)\\mid T_i^\\pi=1]=E[\\mu_0(X_i)\\mid T_i^\\pi=1]$.\n\n\\emph{Proof of Theorem 3 (PDF p.~22).} Writing $\\widehat V(\\pi)=E[w_{\\mathrm{TT}}]/n$, apply Assumption 1 and the tower property directly to the total reward:\n\\[\n\\hl{E[T_i^\\pi\\hat\\mu_0(X_i)]=E\\bigl[E[T_i^\\pi\\hat\\mu_0(X_i)\\mid X_i,U_i]\\bigr]=E[\\pi(X_i,U_i)\\mu_0(X_i)].}\n\\]\nIn the preceding regret inequality, use $\\hl{\\widehat V(\\pi)-V(\\pi)=E[\\pi(X_i,U_i)(\\mu_0(X_i,U_i)-\\mu_0(X_i))]}$. Jensen's inequality and Assumption 2 then give the printed bound.\n\nThe treated residual $T_i^\\pi(\\hat\\mu_0(X_i)-\\mu_0(X_i))$ is assumed absolutely integrable for every feasible policy and unit, so its conditional expectation is an ordinary conditional expectation. For the weighted identity used in Theorems 1 and 2, assume also that $g(X_i)\\tau(X_i)$ and each individual reward summand $(Y_i-\\hat\\mu_0(X_i))T_i^\\pi g(X_i)$ are absolutely integrable, for every applicable weight, feasible policy and unit. Finiteness of the expectation of the total weighted reward alone does not suffice: cancellations between nonintegrable summands can invalidate the weighted identity and the optimality asserted by Theorem 1. Integrability of the estimator at untreated evaluations is not required.\n\nPropositions 2, 3 and 9 and Theorems 1, 2 and 3 are read with this Assumption 1 in place of the printed one, with these additional weighted integrability conditions for Theorems 1 and 2.", "evidence": "I use exact counterexamples to the printed implication, computed in rational arithmetic, and derive the total-effect identities and regret bound under the corrected assumption.\n\n\\emph{Example A.} Let $n=1$, let $X_1$ take the values 0 and 1 with probability $1/2$ each, and let $Y_1(0)=0$ and $Y_1(1)=1/2$. Then $\\mu_0(x)=0$ and $\\tau(x)=1/2$ for both values of $x$. Define the estimator by $\\hat\\mu_0(x)=2\\cdot\\mathbf{1}\\{x=X_1\\}-1$. Note that it uses only the covariate, which the principal observes, and that it does not depend on the policy.\n\\begin{audittable}{ccccc}{Example A: the sampling distribution of the estimator.}\nProbability & $X_1$ & $\\hat\\mu_0(0)$ & $\\hat\\mu_0(1)$ & $\\hat\\mu_0(X_1)$ \\\\\n$1/2$ & 0 & $1$ & $-1$ & $1$ \\\\\n$1/2$ & 1 & $-1$ & $1$ & $1$ \\\\\n\\end{audittable}\nFor each fixed $x$, I have $E[\\hat\\mu_0(x)]=(1/2)(1)+(1/2)(-1)=0=\\mu_0(x)$, and Assumption 1 holds. However, $\\hat\\mu_0(X_1)=1$ in both states, so under the treat-all policy $E[\\hat\\mu_0(X_1)\\mid X_1,T_1^\\pi=1]=1$, not $\\mu_0(X_1)=0$. For a policy with $\\pi(0)=p_0$ and $\\pi(1)=p_1$, a direct computation gives\n\\[\nE[w_{\\mathrm{TT}}]=-\\tfrac14(p_0+p_1),\\qquad V(\\pi)=\\tfrac14(p_0+p_1).\n\\]\nThe reward is maximized only at $p_0=p_1=0$, and welfare only at $p_0=p_1=1$. The best response has regret $1/2$, which contradicts Propositions 3 and 9, and Theorem 1 with $g\\equiv1$. Since $n=1$, the rewards $w_{\\mathrm{ATT}}$ and $w_{\\mathrm{TT}}$ coincide, and the best response stated in Proposition 7 also fails. If $U_1$ is constant, then $\\gamma_{\\mathrm{marg}}=0$, and the bound of Theorem 3 would read $1/2\\le0$.\n\\begin{audittable}{cccc}{Example A: expected reward and welfare of the four deterministic policies. The last column replaces $X_1$ in the estimator by an independent copy $Z$ of the covariate.}\nPolicy $(p_0,p_1)$ & $E[w_{\\mathrm{TT}}]$, estimator from $X_1$ & Welfare $V(\\pi)$ & $E[w_{\\mathrm{TT}}]$, estimator from $Z$ \\\\\n$(0,0)$ & $0$ & $0$ & $0$ \\\\\n$(0,1)$ & $-1/4$ & $1/4$ & $1/4$ \\\\\n$(1,0)$ & $-1/4$ & $1/4$ & $1/4$ \\\\\n$(1,1)$ & $-1/2$ & $1/2$ & $1/2$ \\\\\n\\end{audittable}\n\n\\emph{Example B.} In the setting of Example A, I change two signs: $Y_1(1)=-1/2$ and $\\hat\\mu_0(x)=1-2\\cdot\\mathbf{1}\\{x=X_1\\}$. The estimator is again unbiased at each fixed $x$, and it equals $-1$ at $X_1$. Then $E[w_{\\mathrm{TT}}]=E[w_{\\mathrm{ATT}}]=(p_0+p_1)/4$ and $V(\\pi)=-(p_0+p_1)/4$. The best response treats everyone, and its regret $1/2$ exceeds $\\max_{\\pi}V(\\pi)=0$, which contradicts Proposition 2.\n\n\\emph{Example C.} Let $j$ and $k$ be two agents with one unit each. Their covariates $X_j$ and $X_k$ are independent and take the values 0 and 1 with probability $1/2$ each, as does the reference covariate, so that all weights $g$ equal 1. The untreated outcomes are 0, and the treated outcomes are $1/4$ for agent $j$ and $1/8$ for agent $k$. Both agents are scored with the estimator $\\hat\\mu_0(x)=\\mathbf{1}\\{x=X_j\\}-\\mathbf{1}\\{x=X_k\\}$, which has mean 0 at each fixed $x$. Then $\\max_{\\pi_j}E[w_j]=0$ and $\\max_{\\pi_k}E[w_k]=5/8$, although $\\tau^j(x)>\\tau^k(x)>0$ for every $x$. The ranking of Theorem 2 is reversed.\n\n\\emph{The corrected assumption.} Suppose that $E[\\hat\\mu_0(X_i)\\mid X_i,U_i,T_i^\\pi=1]=\\mu_0(X_i)$ whenever $\\pi(X_i,U_i)>0$. Since $P(T_i^\\pi=1\\mid X_i,U_i)=\\pi(X_i,U_i)$, I have\n\\[\nE[T_i^\\pi\\hat\\mu_0(X_i)\\mid X_i,U_i]=\\pi(X_i,U_i)\\,\\mu_0(X_i),\n\\]\nand ignorability and consistency imply that $E[T_i^\\pi Y_i\\mid X_i,U_i]=\\pi(X_i,U_i)\\,\\mu_1(X_i,U_i)$. Subtracting, summing over $i$ and taking expectations, I obtain\n\\[\n\\tfrac1n E[w_{\\mathrm{TT}}]=E\\bigl[\\pi(X_i,U_i)\\bigl(\\mu_1(X_i,U_i)-\\mu_0(X_i)\\bigr)\\bigr].\n\\]\nWithout $U_i$ the right side is $E[\\pi(X_i)\\tau(X_i)]=V(\\pi)$. Reward and welfare then have the same maximizers over any feasible set, which is Proposition 3. The following weighted identity uses the per-unit integrability conditions stated below. With a weight $g(X_i)$, I find in the same way that $E[w^g_{\\mathrm{TT}}]=nE[g(X_i)\\pi(X_i)\\tau(X_i)]$, which is the identity used for Theorems 1 and 2. With $U_i$, I write $\\widehat V(\\pi)=E[w_{\\mathrm{TT}}]/n$. Then\n\\[\n|\\widehat V(\\pi)-V(\\pi)|=\\bigl|E\\bigl[\\pi(X_i,U_i)\\bigl(\\mu_0(X_i,U_i)-\\mu_0(X_i)\\bigr)\\bigr]\\bigr|\\le E\\bigl|\\mu_0(X_i,U_i)-\\mu_0(X_i)\\bigr|\\le\\gamma_{\\mathrm{marg}}.\n\\]\nLet $\\pi^*$ maximize welfare and $\\pi^w$ maximize expected reward. Since $\\widehat V(\\pi^*)\\le\\widehat V(\\pi^w)$, I conclude that\n\\[\nV(\\pi^*)-V(\\pi^w)=\\bigl[V(\\pi^*)-\\widehat V(\\pi^*)\\bigr]+\\bigl[\\widehat V(\\pi^*)-\\widehat V(\\pi^w)\\bigr]+\\bigl[\\widehat V(\\pi^w)-V(\\pi^w)\\bigr]\\le2\\gamma_{\\mathrm{marg}},\n\\]\nwhich is the bound of Theorem 3. If $\\hat\\mu_0$ is unbiased at each fixed $x$ and independent of the agent's covariates, private information and treatment randomization, then $E[\\hat\\mu_0(X_i)\\mid X_i=x,U_i=u,T_i^\\pi=1]=E[\\hat\\mu_0(x)]=\\mu_0(x)$, and the corrected assumption holds. The last column of the second table shows this for Example A.", "problem": "Assumption 1 fixes the mean of the random function $\\hat\\mu_0$ at each fixed argument $x$. The proofs use the mean of $\\hat\\mu_0$ at the agent's own treated observations, that is, of $\\hat\\mu_0(X_i)$ given $T_i^\\pi=1$. The first does not imply the second when the estimator depends on the sample on which the agent is scored. In that case $\\hat\\mu_0(X_i)$ can be biased for $\\mu_0(X_i)$ although $\\hat\\mu_0(x)$ is unbiased for every fixed $x$, so the three highlighted steps need not hold. If Assumption 1 is the only condition on the estimator, then the conclusions drawn from these steps fail as well: the best responses of Propositions 7 and 9, the bound of Proposition 2, the zero regret of Proposition 3 and Theorem 1, the ranking property of Theorem 2, and the bound $2\\gamma_{\\mathrm{marg}}$ of Theorem 3.\n\nThe paper does not exclude such estimators. The auxiliary data on PDF p.~4 are introduced as ``a concrete example'', and Appendix A.1, item 2, proposes to add the agent's own untreated units to the estimate of $\\mu_0$. Assumption 1 is a legitimate assumption. What fails is the step from it to the conditional statements.\n\nThe examples below do not require an agent who observes the realized estimate before choosing a policy. In each of them the policy is a fixed function of the covariates, as in Section 2, and the expected reward is taken over the estimator as well.", "what_survives": "Without information asymmetry, the corrected assumption, with integrable treated estimator residuals, gives the identity $E[w_{\\mathrm{TT}}]=nV(\\pi)$ and the zero regret of Proposition 3 for any feasible set. With the weighted integrability conditions below, it also gives Theorem 1 for unconstrained policies and positive weights, by the derivation above. Under information asymmetry, it gives the bound $2\\gamma_{\\mathrm{marg}}$ of Theorem 3. The corrected assumption restricts the joint distribution of the estimator and the agent's sample. An unbiased estimator from independent auxiliary data, as in the example on PDF p.~4 and in Appendix D.2, satisfies it. An estimator that reuses the agent's sample need not. The rewards for the average and the total treated outcome do not use the estimator. The experiments of Section 6 take the fitted means as given and do not simulate estimation of a separate baseline, so no numerical correction follows from this defect. This correction concerns only the replacement of $\\hat\\mu_0$ by $\\mu_0$. The best responses to the average rewards are corrected in \\fid{P006-E001}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E002", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of Assumption 3 (Non-degenerate w) and Theorem 4. Two red boxes mark, in the theorem, the words 'Suppose w satisfies Assumption 3.' and the words 'then regret is unbounded.'", "caption": "Appendix A.2, PDF p. 14: Assumption 3 and Theorem 4. The marked hypothesis places no condition on the feasible set $\\Pi$, and the marked conclusion does not say for which best response the regret is unbounded.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E003.png" }, { "alt": "Crop of the second half of the proof of Theorem 4, from 'Denote the welfare maximizing policy for distribution D as' to the last line. A red box marks the sentence 'Then the set of welfare maximizing policies contains a pi star with pi star of x bar equal to 0 for all x bar in X bar.' The regret display below it ends with 'greater than or equal to minus delta times (alpha minus beta) times rho'.", "caption": "Proof of Theorem 4, PDF p. 14. The marked step takes a welfare-maximizing policy that withholds treatment on $\\bar{\\mathcal{X}}$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E003-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T10", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T10" }, { "id": "P006-T11", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T11" } ], "headline": "Adding feasibility of the policy that treats nobody repairs Theorem 4, with the conclusion that some best response has arbitrarily large regret; the printed proof assumes that treatment can be withheld.", "id": "P006-E003", "location": "Appendix A.2, Assumption 3, Theorem 4 and its proof, PDF p. 14", "pdf_page": 14, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T10", "P006-T11" ], "text": { "as_printed": "\\textbf{Theorem 4.} \\hl{Suppose $w$ satisfies Assumption 3.} Let $\\mathcal{S}$ be the set of all pairs of distributions $\\mathcal{D},\\mathcal{D}'$ such that the distributions of $X_i,Y_i(1)$ remain unchanged between $\\mathcal{D}$ and $\\mathcal{D}'$. If for all pairs of distributions $\\mathcal{D},\\mathcal{D}'\\in\\mathcal{S}$, $E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=E_{\\mathcal{D}'}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ for all $\\pi$, \\hl{then regret is unbounded.}\n\nFrom the proof: ``Choose $\\beta>\\alpha$. \\hl{Then the set of welfare maximizing policies contains a $\\pi^*$ with $\\pi^*(\\bar x)=0$ for all $\\bar x\\in\\bar{\\mathcal{X}}$.} The regret is then\n\\[\nR(\\pi')=E_{\\mathcal{D}}[(\\pi^*(X_i)-\\pi'(X_i))(\\mu_1(X_i)-\\mu_0(X_i))]\\ge-\\delta(\\alpha-\\beta)\\rho.\n\\]\nFixing $\\delta>0,\\rho>0,\\alpha<\\infty$ and choosing $\\beta$ to be arbitrarily high results in arbitrarily high regret.''", "corrected": "\\textbf{Theorem 4.} Suppose \\hl{that the policy $\\pi\\equiv0$ belongs to $\\Pi$ and} that $w$ satisfies Assumption 3. Suppose that, for every $\\pi\\in\\Pi$, the expected reward $E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ takes the same value for all distributions $\\mathcal{D}$ with the same joint distribution of $(X_i,Y_i(1))$, all expectations being finite. Then \\hl{for every $M>0$ there exist a distribution $\\mathcal{D}$ and a best response $\\pi_*\\in\\arg\\max_{\\pi\\in\\Pi}E_{\\mathcal{D}}[w(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]$ such that $R_{\\mathcal{D}}(\\pi_*)>M$.}\n\nIn the proof, the steps from ``Let $\\pi'\\in\\arg\\max$'' to the regret display are replaced by the construction with $Y(0)=Y(1)+B$, which gives $R_{\\mathcal{D}_B}(\\pi_*)=B\\,E[\\pi_*(X)]$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. Let $n=1$, let $X=0$ with probability one, let $\\Pi$ consist of the single policy $\\pi(0)=1$, and let $w=w_{\\mathrm{ATO}}$. For $B>0$, let $Y(1)=1$ and $Y(0)=1+B$. Assumption 3 holds because the only feasible policy treats with probability $1$. For $n=1$, I have $E[w_{\\mathrm{ATO}}]=E[\\pi(X)Y(1)]$, which does not involve $Y(0)$. The expected reward is then unchanged when the distribution of $Y(0)$ changes. The welfare of the only feasible policy is $-B$, which is also the maximum of the welfare over $\\Pi$, and I find that the regret is $0$ for every $B$. With the constants of the printed proof, $\\bar{\\mathcal{X}}=\\{0\\}$, $\\rho=\\delta=\\alpha=1$ and $\\beta=1+B$, the printed bound reads $0\\ge B$.\n\nFor the corrected statement, by Assumption 3 there exist a distribution $\\mathcal{D}_*$ and a best response $\\pi_*$ such that $c=E_{\\mathcal{D}_*}[\\pi_*(X)]>0$. For $B>0$, let $\\mathcal{D}_B$ be the distribution with the same joint distribution of $(X,Y(1))$ as $\\mathcal{D}_*$ and with $Y(0)=Y(1)+B$. Since the expected reward of each feasible policy is the same under $\\mathcal{D}_B$ and $\\mathcal{D}_*$, the policy $\\pi_*$ is a best response under $\\mathcal{D}_B$. Under $\\mathcal{D}_B$, the treatment effect is $\\tau(x)=-B$ for all $x$. Then every feasible $\\pi$ has welfare $V(\\pi)=-B\\,E[\\pi(X)]\\le0$, and the policy that treats nobody attains $0$. I obtain\n\\[\nR_{\\mathcal{D}_B}(\\pi_*)=0-(-Bc)=Bc,\n\\]\nand for $B=(M+1)/c$ I have $R_{\\mathcal{D}_B}(\\pi_*)>M$. Note that the conclusion cannot be extended to every best response: with $w=0$ and $\\Pi$ consisting of the policy that treats nobody and the policy that treats everyone, both policies are best responses, and for $Y(1)=1$ and $Y(0)=1+B$ the first has regret $0$ and the second has regret $B$.", "problem": "Section 2 measures regret against the best policy in the feasible set $\\Pi$, and Theorem 4 places no condition on $\\Pi$. Its proof takes a welfare-maximizing policy $\\pi^*$ that does not treat on the set $\\bar{\\mathcal{X}}$. Such a policy exists only if $\\Pi$ contains a policy that withholds treatment there. If $\\Pi$ contains only the policy that treats everyone, then that policy is both the best response and the welfare maximizer, and the regret is $0$ for every distribution. The proof also takes $\\pi'$ and $\\mu_1$ to be constant on $\\bar{\\mathcal{X}}$, and Assumption 3 does not supply a set of positive probability on which they are constant. The conclusion is written for the regret of a best response without saying which one, and when several policies tie, some of them can have regret $0$.", "what_survives": "Proposition 1 assumes that $\\Pi$ is the set of all functions $\\pi:\\mathcal{X}\\to[0,1]$, and its conclusion that the regret under $w_{\\mathrm{ATO}}$ can be arbitrarily large is not affected. No other result in the paper uses Theorem 4 or Assumption 3. What changes is the scope of Theorem 4 and of the two paragraphs that introduce it in Appendix A.2. The theorem does not cover a feasible set that requires some treatment, such as a fixed positive share of treated units, and it asserts large regret for some best response and not for all of them." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E003", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of six lines of the left column. Two lines end a paragraph: bution P sub X sub 0 satisfies both ranking properties in Definitions 1 and 2. Then: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript k be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times d P sub X sub 0 over d P sub X superscript k be the normalized Radon-Nikodym derivative of the reference distribution P sub X sub 0 with respect. Red outlines surround P sub X superscript k and the words be absolutely continuous with respect to P sub X sub 0.", "caption": "Theorem 2, PDF p. 6, foot of the left column; the statement continues at the top of the right column. The outlines mark the condition of absolute continuity.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E005.png" }, { "alt": "Crop of three lines. The text reads: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript (k) be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times d P sub X sub 0 over d P sub X superscript (k) be the normalized Radon-Nikodym derivative of the reference distribution P sub X sub 0 with respect to agent k's covariate distribution P sub X superscript (k). Then. A red outline surrounds the words from P sub X superscript (k) to P sub X sub 0.", "caption": "Restatement of Theorem 2 in Appendix C.2, PDF p. 20. The outline marks the same condition.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E005-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T5", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T5" }, { "id": "P006-T12", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T12" } ], "headline": "In Theorem 2 the direction of absolute continuity is reversed: the weight $g_k$ is defined when $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$.", "id": "P006-E005", "location": "Theorem 2, PDF p. 6, foot of the left column; restatement in Appendix C.2, PDF p. 20", "pdf_page": 6, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T12", "P006-T5" ], "text": { "as_printed": "\\textbf{Theorem 2} (Ranking Desiderata Satisfied). Let $\\hl{P_{X^k}}$ be absolutely continuous with respect to $\\hl{P_{X_0}}$, and let $g_k=\\frac{1}{n_k}\\frac{dP_{X_0}}{dP_{X^k}}$ be the normalized Radon--Nikodym derivative of the reference distribution $P_{X_0}$ with respect to agent $k$'s covariate distribution $P_{X^k}$. Then setting $w_k$ to be $w_{\\mathrm{TT}}^{g_k}$ for agent $k$'s treatment population satisfies both ranking properties in Definitions 1 and 2 as long as $\\Pi_k$ is unconstrained and treatment effects are nonnegative, $\\tau^k(x)\\geq0$, for all $k\\in\\{1,\\ldots,K\\}$.", "corrected": "\\textbf{Theorem 2} (Ranking Desiderata Satisfied). Let $\\hl{P_{X_0}}$ be absolutely continuous with respect to $\\hl{P_{X^k}}$, and let $g_k=\\frac{1}{n_k}\\frac{dP_{X_0}}{dP_{X^k}}$ be the normalized Radon--Nikodym derivative of the reference distribution $P_{X_0}$ with respect to agent $k$'s covariate distribution $P_{X^k}$. Then setting $w_k$ to be $w_{\\mathrm{TT}}^{g_k}$ for agent $k$'s treatment population satisfies both ranking properties in Definitions 1 and 2 as long as $\\Pi_k$ is unconstrained and treatment effects are nonnegative, $\\tau^k(x)\\geq0$, for all $k\\in\\{1,\\ldots,K\\}$.\n\nThe same change applies to the restatement on PDF p.~20, where the distribution is written $P_{X^{(k)}}$. If the weights $g_k$ must also be strictly positive almost everywhere under $P_{X^k}$, as the proof of Theorem 1 assumes of its weight, then the two distributions must be mutually absolutely continuous. The ranking conclusion does not need this.", "evidence": "The evidence is an exact counterexample to the sufficiency of the printed condition, and a derivation of the theorem under the corrected one.\n\n\\emph{Counterexample.} On $[0,1]$, let $L$ be Lebesgue measure, and let $P_{X^k}=L$ and $P_{X_0}=(L+\\delta_{1/2})/2$. For every Borel set $C$, I have $P_{X_0}(C)\\ge P_{X^k}(C)/2$, which implies that $P_{X^k}$ is absolutely continuous with respect to $P_{X_0}$, as the theorem assumes. Both distributions have support $[0,1]$. However, $P_{X^k}(\\{1/2\\})=0$ and $P_{X_0}(\\{1/2\\})=1/2$. If a density $f=dP_{X_0}/dP_{X^k}$ existed, then I would have $1/2=P_{X_0}(\\{1/2\\})=\\int_{\\{1/2\\}}f\\,dP_{X^k}=0$. The example shows that the printed condition does not suffice for the construction. It is not a counterexample to the ranking conclusion in a case where the derivative exists.\n\n\\emph{Derivation.} Suppose that $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$, and write $f=dP_{X_0}/dP_{X^k}$, so that $g_k=f/n_k$. Then $\\int h\\,dP_{X_0}=\\int hf\\,dP_{X^k}$ for every nonnegative measurable function $h$. Using the expected score of Theorem 1 and this identity with $h=\\tau^k\\pi_k$, I obtain\n\\[\nE[w_k]=n_k\\int\\tau^k(x)\\pi_k(x)g_k(x)\\,dP_{X^k}(x)=\\int\\tau^k(x)\\pi_k(x)\\,dP_{X_0}(x).\n\\]\nSince $0\\le\\pi_k\\le1$ and $\\tau^k\\ge0$, the maximum over $\\pi_k$ is $E[\\tau^k(X_0)]$, attained at $\\pi_k\\equiv1$. If $\\tau^j(x)\\ge\\tau^k(x)$ for all $x$, then $E[\\tau^j(X_0)]\\ge E[\\tau^k(X_0)]$, and Definition 1 holds. The hypothesis of Definition 2 is this inequality itself.", "problem": "The derivative $dP_{X_0}/dP_{X^k}$ of the whole reference distribution exists when $P_{X_0}$ is absolutely continuous with respect to $P_{X^k}$. The theorem assumes the opposite direction, which does not guarantee that the weight $g_k$ is defined. Section 4 assumes that all covariate distributions have the same support $\\mathcal X$. If support is understood in the topological sense, this assumption does not supply the missing direction. The intended condition is clear from the derivative that the theorem names, and with it the change-of-measure argument is valid.", "what_survives": "The ranking conclusion and its proof are unchanged once the condition is reversed. Theorem 1, the results of Sections 3 and 5 and the experiments of Section 6 do not use the condition. Section 6 contains no experiment with reweighting to a reference distribution. The expected score used in the derivation requires the corrected Assumption 1 of \\fid{P006-E002}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E005", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the first paragraph of Appendix D.3, six lines, beginning 'Suppose the principal estimates mu hat 0 of x from the untreated data from the agent's treatment population'. A red box marks 'pi of x greater than 0 for all x in X' after the words 'maintains positivity'.", "caption": "Appendix D.3, first paragraph, PDF p. 23. The marked condition is stated as positivity for identifying the untreated outcome from the agent’s untreated units.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006.png" }, { "alt": "Crop of the statement of Theorem 5 with its display 'R of pi w is at most 2 gamma max'. A red box marks the hypothesis 'pi of x, u greater than 0 for all x in X, u in U'.", "caption": "Theorem 5, PDF p. 23. The marked hypothesis requires a positive probability of treatment.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006-b.png" }, { "alt": "Crop of the end of the proof of Theorem 5 and the statement of Lemma 3. A red box marks 'pi of x, u greater than 0 for all x in X, u in U' in the sentence 'Since for all pi in Pi, ..., we can apply Lemma 3 below'. Lemma 3 begins 'Suppose that for all pi in Pi, P of U given X equals x and T pi equals 0 is a well defined probability distribution.'", "caption": "Proof of Theorem 5, PDF p. 24. The marked condition is given as the reason that Lemma 3 applies. Lemma 3, at the bottom of the crop, assumes that the distribution of $U$ given $X=x$ and $T^\\pi=0$ is well defined.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E006-c.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T15", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T15" }, { "id": "P006-T16", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T16" } ], "headline": "The positivity condition of Appendix D.3 and Theorem 5 is that untreated units occur with positive probability at each $x$, $P(T^\\pi=0\\mid X=x)>0$, for which $\\pi(x,u)<1$ suffices; the printed condition $\\pi>0$ concerns the treated units.", "id": "P006-E006", "location": "Appendix D.3, first paragraph and Theorem 5, PDF p. 23; proof of Theorem 5, sentence before Lemma 3, PDF p. 24", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T15", "P006-T16" ], "text": { "as_printed": "Appendix D.3, first paragraph, PDF p.~23: ``As discussed in Sections 5 and A, under full information symmetry, the mean conditional untreated potential outcome is identifiable as long as the agent's treatment rule class $\\Pi$ maintains positivity, \\hl{$\\pi(x)>0$} for all $x\\in\\mathcal{X}$.''\n\nPDF p.~23: ``\\textbf{Theorem 5.} Suppose for all $\\pi\\in\\Pi$, \\hl{$\\pi(x,u)>0$} for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$. If the principal applies the reward function $w_{\\mathrm{TT}}$ using an estimate $\\hat\\mu_0^\\pi(x)$, then under Assumption 4, the regret is upper bounded as $R(\\pi^w)\\le2\\gamma_{\\max}$.''\n\nProof, PDF p.~24: ``Since for all $\\pi\\in\\Pi$, \\hl{$\\pi(x,u)>0$} for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$, we can apply Lemma 3 below, which says that $\\hat\\mu_0^\\pi(x)\\in[\\min_u\\mu_0(x,u),\\max_u\\mu_0(x,u)]$.''", "corrected": "Appendix D.3, first paragraph: ``\\ldots\\ the mean conditional untreated potential outcome is identifiable as long as the agent's treatment rule class $\\Pi$ maintains positivity, \\hl{$\\pi(x)<1$} for all $x\\in\\mathcal{X}$.''\n\n\\textbf{Theorem 5.} Suppose for all $\\pi\\in\\Pi$, \\hl{$P(T^\\pi=0\\mid X=x)=E[1-\\pi(x,U)\\mid X=x]>0$ for almost every $x$ with respect to the distribution of $X$}. If the principal applies the reward function $w_{\\mathrm{TT}}$ using the estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$, then under Assumption 4, the regret is upper bounded as $R(\\pi^w)\\le2\\gamma_{\\max}$. The hypothesis holds if $\\pi(x,u)<1$ for all $x\\in\\mathcal{X}$, $u\\in\\mathcal{U}$.\n\nProof: ``Since for all $\\pi\\in\\Pi$, \\hl{$P(T^\\pi=0\\mid X=x)>0$ for almost every $x$ with respect to the distribution of $X$}, the distribution $P_{U\\mid X=x,T^\\pi=0}$ is well defined and we can apply Lemma 3 below \\ldots''", "evidence": "The evidence is an exact counterexample to the printed hypothesis and a derivation of the bound under the corrected one. Let $X=0$ and $U=0$ with probability one, let $\\Pi$ consist of the single policy $\\pi(0,0)=1$, and let $Y(1)=1$. Consider $Y(0)=0$ and $Y(0)=2$. In both cases $\\pi>0$, Assumption 4 holds with $\\gamma_{\\max}=0$, and the observed data are the single point $(X,T,Y)=(0,1,1)$. The untreated means are $0$ and $2$, and the untreated mean is not identified. Further, $P(T^\\pi=0\\mid X=0)=0$, and the conditional mean in $\\hat\\mu_0^\\pi(0)$ is the ratio $0/0$. Untreated units elsewhere do not help: with $P(X=0)=P(X=1)=1/2$, $\\pi(0)=1$ and $\\pi(1)=1/2$, I have $P(T^\\pi=0)=1/4$ and $P(T^\\pi=0\\mid X=0)=0$.\n\nFor the corrected hypothesis, I write $q_\\pi(x)=P(T^\\pi=0\\mid X=x)=E[1-\\pi(x,U)\\mid X=x]$ and suppose that $q_\\pi(x)>0$. Then the distribution of $U$ given $X=x$ and $T^\\pi=0$ has density $(1-\\pi(x,u))/q_\\pi(x)$ with respect to the distribution of $U$ given $X=x$, and $\\hat\\mu_0^\\pi(x)$ is the mean of $\\mu_0(x,\\cdot)$ under it. This implies that $|\\hat\\mu_0^\\pi(x)-\\mu_0(x,u)|\\le\\Delta(x,u)$ for all $u$, where $\\Delta(x,u)=\\max_{\\tilde u\\in\\mathcal{U}}|\\mu_0(x,u)-\\mu_0(x,\\tilde u)|$ is the quantity defined in Assumption 4. Let $S(\\pi)=E[\\pi(X,U)(\\mu_1(X,U)-\\hat\\mu_0^\\pi(X))]$ denote the expected reward per unit. Since $0\\le\\pi\\le1$, I have $|S(\\pi)-V(\\pi)|\\le E[\\Delta(X,U)]\\le\\gamma_{\\max}$ for every $\\pi\\in\\Pi$. Let $\\pi^*$ maximize $V$ over $\\Pi$. Since $\\pi^w$ maximizes $S$ over $\\Pi$, I obtain\n\\[\nR(\\pi^w)=[V(\\pi^*)-S(\\pi^*)]+[S(\\pi^*)-S(\\pi^w)]+[S(\\pi^w)-V(\\pi^w)]\\le2\\gamma_{\\max}.\n\\]", "problem": "In the paper $\\pi$ is the probability of treatment. The estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$ is a mean over untreated units. It is defined only if untreated units occur with positive probability at $x$, and Lemma 3 assumes that the distribution of $U$ given $X=x$ and $T^\\pi=0$ is well defined. The printed condition $\\pi>0$ ensures instead that treated units occur, and it allows a policy that treats everyone. The condition is printed in this form in three places. Appendix A.1 states the condition correctly, as $P(T_i^\\pi=0\\mid X_i=x)>0$, and notes that it can be enforced by restricting $\\Pi$ to treatment rules with $\\pi(x)<1$.", "what_survives": "The bound $R(\\pi^w)\\le2\\gamma_{\\max}$ of Theorem 5 holds under the corrected hypothesis, with the same constant, and Lemma 3 holds under its own hypothesis. The results of Appendix D.2, Theorem 3 and Proposition 5 do not use the condition. Appendix E.3 states that the experiments do not simulate the estimation of $\\hat\\mu_0$ from a subset of the data, and no reported number depends on the condition. The corrected hypothesis restricts $\\Pi$: it excludes every policy that treats all units at a value of $x$ of positive probability. Appendix A.1 notes that such a restriction may not always be possible or ethical." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E006", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the six-line paragraph beginning 'Assumption 4 is stronger than Assumption 2'. In its last sentence, two red boxes mark the words 'Assumption 4' after 'Intuitively,' and the words 'allocate treatment to single values of u tilde in U'.", "caption": "Appendix D.3, paragraph after Assumption 4, PDF p. 23. The first mark is the assumption named in the sentence beginning “Intuitively”. The second mark is the description of the agent’s policy.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E014.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T17", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T17" } ], "headline": "The sentence beginning “Intuitively” refers to Assumption 2 and not Assumption 4, and the agent concentrates the estimate $\\hat\\mu_0^\\pi(x)$ on one type $\\tilde u$ by leaving only that type untreated, not by treating only that type.", "id": "P006-E014", "location": "Appendix D.3, paragraph after Assumption 4, sentence beginning “Intuitively”, PDF p. 23", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T17" ], "text": { "as_printed": "Assumption 4 is stronger than Assumption 2 in the sense that Assumption 2 is not sufficient to bound the regret when the principal estimates the mean conditional untreated potential outcome from the agent's data using $\\hat\\mu_0^\\pi(X)$. Furthermore, the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$. Intuitively, \\hl{Assumption 4} is not sufficient to bound the regret when the principal uses $\\hat\\mu_0^\\pi(X)$ because the agent can choose $\\pi$ to \\hl{allocate treatment to single values of $\\tilde u\\in\\mathcal{U}$}, such that $\\hat\\mu_0^\\pi(x)$ ends up matching a single value $\\mu_0(x,\\tilde u)$.", "corrected": "Intuitively, \\hl{Assumption 2} is not sufficient to bound the regret when the principal uses $\\hat\\mu_0^\\pi(X)$, because the agent can \\hl{leave only a selected type $\\tilde u$ of positive conditional probability untreated}, so that $\\hat\\mu_0^\\pi(x)=\\mu_0(x,\\tilde u)$.", "evidence": "This is a check of the page and a short computation. If $U$ takes finitely many values with conditional probabilities $p_u$ given $X=x$, then $\\hat\\mu_0^\\pi(x)=\\sum_u\\mu_0(x,u)p_u(1-\\pi(x,u))/\\sum_up_u(1-\\pi(x,u))$. For three equally likely types with $\\mu_0(x,u)$ equal to $0$, $1$ and $3$, I find $\\hat\\mu_0^\\pi(x)=2$ when only the first type is treated, and $\\hat\\mu_0^\\pi(x)=0$ when only the first type is left untreated.", "problem": "The sentence means Assumption 2: the preceding sentence says that Assumption 2 is not sufficient, and Theorem 5, which follows, shows that Assumption 4 is sufficient. Since the estimate $\\hat\\mu_0^\\pi(x)=E[Y_i\\mid T^\\pi=0,X=x]$ is a mean over the untreated units, it equals $\\mu_0(x,\\tilde u)$ when the type $\\tilde u$ is the only one left untreated, not when it is the only one treated.", "what_survives": "No result depends on this sentence; the hypothesis of Theorem 5 is the subject of \\fid{P006-E006}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E014", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the paragraph 'A second failure mode would be if alpha equals minus 1, and beta is greater than 1. Then the regret is given by', the display, and the three lines that follow. A red box marks the right part of the display, 'minus p times (beta minus 1) equals 1 minus p beta', which follows 'R of pi w equals (1 minus p)'.", "caption": "Appendix B.4, proof of Proposition 1, PDF p. 17: the second failure mode. The marked terms are the welfare of the agent’s policy, entered with the wrong sign, and the resulting value of the regret.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E007.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T18", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T18" }, { "id": "P006-T19", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T19" } ], "headline": "The regret in the second failure mode is $(1-p)-p(1-\\beta)=1-2p+p\\beta$; the printed display subtracts $p(\\beta-1)$ and has the value $1-p\\beta$.", "id": "P006-E007", "location": "Appendix B.4, proof of Proposition 1, second failure mode, display after “Then the regret is given by”, PDF p. 17", "pdf_page": 17, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T18", "P006-T19" ], "text": { "as_printed": "A second failure mode would be if $\\alpha=-1$, and $\\beta>1$. Then the regret is given by\n\\[\nR(\\pi^w)=(1-p)\\hl{{}-p(\\beta-1)=1-p\\beta}.\n\\]\nThis is also unbounded for unbounded $\\beta$.", "corrected": "Let $\\mathcal{X}=\\{0,1\\}$ with $P(X_i=1)=p$, \\hl{$01$, and let $\\pi^w(0)=0$ and $\\pi^w(1)=1$. Then the regret is given by\n\\[\nR(\\pi^w)=(1-p)-p(\\hl{1-\\beta})=\\hl{1-2p+p\\beta}.\n\\]\nThis is also unbounded for unbounded $\\beta$.", "evidence": "The evidence is a derivation and an exact example. With $\\alpha=-1$ and $\\beta>1$, the treatment effects are $\\tau(0)=0-(-1)=1$ and $\\tau(1)=1-\\beta<0$, which yields $V(\\pi)=(1-p)\\pi(0)+p(1-\\beta)\\pi(1)$. For every policy,\n\\[\n(1-p)-V(\\pi)=(1-p)(1-\\pi(0))+p(\\beta-1)\\pi(1)\\ge0,\n\\]\nwhich implies that the largest welfare is $1-p$, attained by treating $x=0$ only. The agent's policy has $V(\\pi^w)=p(1-\\beta)$. I obtain\n\\[\nR(\\pi^w)=(1-p)-p(1-\\beta)=1-2p+p\\beta,\n\\]\nwhich exceeds the printed expression by $2p(\\beta-1)>0$. For $p=1/2$ and $\\beta=4$, the largest welfare is $1/2$, the welfare of $\\pi^w$ is $-3/2$, and the regret is $2$; the printed expression equals $-1$. In this example with $n=2$, a direct enumeration shows that $\\pi^w$ maximizes the expected reward under $w_{\\mathrm{ATO}}$ as defined. The example does not depend on the rule corrected in \\fid{P006-E001}.", "problem": "For $01/p$, and a regret is never negative.", "what_survives": "Proposition 1 is unchanged. With $p=1/2$ and $\\beta=2m$, the corrected regret equals $m$. The regret in the second failure mode is then unbounded, and the first failure mode, with regret $-\\alpha(1-p)$, is printed correctly. The replication files compute the regret under $w_{\\mathrm{ATO}}$ as the difference between the largest utility and the utility of the agent's policy, both computed from $\\mu_1-\\mu_0$, and no reported number depends on this display." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E007", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of eleven lines of Appendix E.2.1. The text reads: We removed all examples in which the horse was euthanized, and used the remaining outcome variable as Y, where we set Y sub i = 1 if the horse lived, and Y sub i = -1 if the horse died. After a sentence on the data files it continues: To simulate mu sub t of x, we assume that the outcome distribution takes the parametric form, P(Y(t) = 1 given X = x) = sigma(beta 0 + beta 1 transpose x + beta 2 t + beta 3 transpose x t), where sigma(x) = 1 over (1 + e to the minus x) is the standard logistic function. We trained a logistic regression model of Y on X, T, and the interaction term XT to estimate parameters beta hat, and used the resulting estimate to compute mu sub t of x = sigma(beta hat 0 + beta hat 1 transpose x + beta hat 2 t + beta hat 3 transpose x t). A red outline surrounds this last formula. The crop also shows the AUC and accuracy, the statement that mu sub t of x is taken as synthetic mean conditional potential outcomes, and the beginning of a sentence on clinical validity.", "caption": "Appendix E.2.1 (PDF p. 25). The outline marks the printed formula for $\\mu_t(x)$. Its right side is the survival probability of the display above it, and the second line of the crop codes the outcome as $1$ and $-1$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E008.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T20", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T20" }, { "id": "P006-T21", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T21" } ], "headline": "With survival coded $1$ and death coded $-1$, the fitted conditional mean is $\\mu_t(x) = 2\\sigma(\\cdot) - 1$ and not the survival probability $\\sigma(\\cdot)$ that the appendix prints; the released code uses the correct formula.", "id": "P006-E008", "location": "Appendix E.2.1, paragraph beginning “We trained a logistic regression model” (PDF p. 25)", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T20", "P006-T21" ], "text": { "as_printed": "We removed all examples in which the horse was euthanized, and used the remaining ``outcome'' variable as $Y$, where we set $Y_i = 1$ if the horse lived, and $Y_i = -1$ if the horse died. [\\ldots] To simulate $\\mu_t(x)$, we assume that the outcome distribution takes the parametric form,\n\\[\nP(Y(t) = 1 | X = x) = \\sigma(\\beta_0 + \\beta_1^\\top x + \\beta_2 t + \\beta_3^\\top x t)\n\\]\nwhere $\\sigma(x) = \\frac{1}{1+e^{-x}}$ is the standard logistic function.\n\nWe trained a logistic regression model of $Y$ on $X$, $T$, and the interaction term $XT$ to estimate parameters $\\hat\\beta$, and used the resulting estimate to compute $\\mu_t(x) = \\hl{\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t)}$. [\\ldots] When computing agent best responses and regret, we take this function $\\mu_t(x)$ to be given as synthetic mean conditional potential outcomes.", "corrected": "We trained a logistic regression model of $Y$ on $X$, $T$, and the interaction term $XT$ to estimate parameters $\\hat\\beta$, and used the resulting estimate to compute\n\\[\n\\mu_t(x) = \\hl{2}\\,\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t) \\hl{{} - 1} .\n\\]\nHere $Y_i = 1$ if the horse lived and $Y_i = -1$ if the horse died, and $\\sigma(\\hat\\beta_0 + \\hat\\beta_1^\\top x + \\hat\\beta_2 t + \\hat\\beta_3^\\top x t)$ is the fitted probability $P(Y(t) = 1 \\mid X = x)$ of survival. The two probability equations, in Section~6 (PDF p.~7) and in Appendix~E.2.1, stay as printed, and Section~6 may state the same conversion to the mean.", "evidence": "The evidence is a derivation and an exact counterexample. Fix $t$ and $x$, and write $p_t(x) = P(Y(t) = 1 \\mid X = x)$. Since $Y(t)$ takes only the values $1$ and $-1$, I have\n\\[\n\\mathbb{E}[Y(t) \\mid X = x] = 1 \\cdot p_t(x) + (-1) \\cdot (1 - p_t(x)) = 2 p_t(x) - 1 .\n\\]\nSubstituting the fitted probability into this identity, I obtain the corrected formula. For the counterexample, let $X = 0$ with probability one, and let $(Y(0), Y(1))$ equal $(1, 1)$, $(1, -1)$, $(-1, 1)$ and $(-1, -1)$ with probabilities $1/8$, $3/8$, $1/8$ and $3/8$. This law belongs to the printed model, with $\\beta_0 = 0$, $\\beta_2 = -\\log 3$ and $\\beta_1 = \\beta_3 = 0$: the survival probabilities are $\\sigma(0) = 1/2$ without treatment and $\\sigma(-\\log 3) = 1/4$ with treatment. The means are $\\mu_0 = 0$ and $\\mu_1 = -1/2$, and the printed formula gives $\\mu_1 = 1/4$. In this example the rule $\\mu_1(x) > 0$ treats nobody when the correct mean is used and everybody when the printed one is used. The first response has regret $0$, and the second has regret $1/2$. In the released notebook for the Horse Colic data, both prediction functions return \\texttt{(Y\\_probs*2.) - 1.}, and the optimal utility stored there, $0.146954$, agrees with the entry $0.14695$ of Table~1.", "problem": "The appendix codes $Y$ on the scale $\\{-1, 1\\}$, and Section~2 defines $\\mu_t(X)$ as the conditional mean $\\mathbb{E}[Y_i(t) \\mid X]$. The logistic model specifies the probability of survival, $P(Y(t) = 1 \\mid X = x)$, and the printed sentence sets the mean equal to that probability. For an outcome with the values $1$ and $-1$, the mean is twice the probability minus one. The two agree only if the probability is one, a value that the logistic function does not attain. The released notebook computes the means as $2p - 1$ from the fitted probability $p$. The numbers in the paper are then on the stated scale, and the defect is in the description. If the printed formula were followed, the treatment effects would be half as large, and so would the utility and the regret of a fixed treatment rule. The response to the total-outcome reward $w_{\\mathrm{TO}}$ of the appendix could also change. Its rule $\\mu_1(x) > 0$ means $P(Y(1) = 1 \\mid X = x) > 1/2$ on the stated scale, and it holds for every $x$ when the mean is replaced by the probability.", "what_survives": "The correction changes one formula in the description of the experiment. It adds no assumption and changes no estimand or theorem. The released code already applies it. The Horse Colic entries of Tables~1 and~2 need no change for this correction, and they should not be doubled. The probability equations in Section~6 and in the appendix are correct as printed, and the Stroke Trial model and the theoretical results do not involve this formula." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E008", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [ { "alt": "Crop of eight lines of the right column. The text reads: take these as given to produce mu-zero of x and mu-one of x. The fitted mu-zero of x and mu-one of x show 62 horses benefiting from surgery and 146 being better off without surgery. On the horses that would benefit, the average benefit was 0.147, which is fairly significant. The clinical validity of these estimated potential outcomes cannot be verified from this data alone, and we instead take these estimates as synthetic potential outcomes. Red outlines surround the words the horses that would benefit and the number 0.147.", "caption": "Section 6, paragraph “Horse Colic dataset”, PDF p. 7, right column. The outlines mark the group to which the average is attributed and the number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009.png" }, { "alt": "Crop of five lines of the left column. The text reads: 1360 patients benefiting from heparin and 5904 being better off without heparin. On the patients that would benefit, the average benefit of treatment was 0.025. This tracks with the study's findings that the benefit of heparin was non-significant and inconclusive. Red outlines surround the words the patients that would benefit, which run over a line break, and the number 0.025.", "caption": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 8, top of the left column. The outlines mark the group to which the average is attributed and the number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009-b.png" }, { "alt": "Two bar charts side by side under the title: The printed averages are taken over the whole sample, not over the units that would benefit. Left panel, Horse Colic (surgery): a blue bar of height 0.147 labelled all 255 horses and an orange bar of height 0.604 labelled the 62 horses that would benefit. Right panel, Stroke Trial (heparin): a blue bar of height 0.025 labelled all 7264 patients and an orange bar of height 0.135 labelled the 1360 patients that would benefit. A legend says that blue is the average over the whole analysis sample (the printed number) and orange the average over the units with a positive fitted effect. The vertical axis is the average fitted benefit of treatment in units of the outcome Y.", "caption": "The printed numbers (blue) and the averages over the units that would benefit (orange), from the stored outputs of the released notebooks. The two panels have different vertical scales, since the outcome is $\\pm1$ for the Horse Colic data and a score between $-4$ and $3$ for the Stroke data.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E009-benefit.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T9", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T9" }, { "id": "P006-T28", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T28" } ], "headline": "The numbers 0.147 and 0.025 are averages of the positive fitted effects over all horses and all patients in the analysis samples; among the horses and patients that would benefit, the average fitted benefit is about 0.604 and 0.135.", "historical_evidence_type": "lean_proof", "id": "P006-E009", "location": "Section 6, paragraph “Horse Colic dataset”, PDF p. 7, right column; paragraph “International Stroke Trial dataset”, PDF p. 8, top of the left column", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T28", "P006-T9" ], "text": { "as_printed": "\\emph{PDF p.~7.} The fitted $\\mu_0(x)$ and $\\mu_1(x)$ show 62 horses benefiting from surgery and 146 being better off without surgery. On \\hl{the horses that would benefit}, the average benefit was \\hl{0.147}, which is fairly significant.\n\n\\emph{PDF pp.~7--8.} The fitted $\\mu_0(x)$ and $\\mu_1(x)$ showed 1360 patients benefiting from heparin and 5904 being better off without heparin. On \\hl{the patients that would benefit}, the average benefit of treatment was \\hl{0.025}. This tracks with the study's findings that the benefit of heparin was non-significant and inconclusive.", "corrected": "\\emph{PDF p.~7.} \\ldots \\hl{Under the fitted model, treating only the horses with positive fitted effects gives an average welfare gain of} 0.147 \\hl{over all 255 horses in the analysis sample, with zero contributions from the horses not treated.}\n\n\\emph{PDF pp.~7--8.} \\ldots \\hl{Under the fitted model, treating only the patients with positive fitted effects gives an average welfare gain of} 0.025 \\hl{over all 7264 patients in the analysis sample, with zero contributions from the patients not treated.}\n\nAlternatively, the two sentences keep their wording and the numbers become \\hl{0.604} and \\hl{0.135}. In general, if $r\\hl{>0}$ is the fraction of units with a positive fitted effect and $V^*$ is the \\hl{unconstrained} optimal utility, then the mean fitted benefit among those units is $V^*/r$.", "evidence": "The evidence is a recalculation from the stored outputs of the released notebooks, in exact arithmetic. Let $\\hat\\tau_i=\\mu_1(X_i)-\\mu_0(X_i)$ for the $n$ units of the analysis sample, let $k$ be the number of units with $\\hat\\tau_i>0$, and let $S$ be the sum of $\\hat\\tau_i$ over those units. The notebooks compute $V=S/n$. The average over the units that would benefit is\n\\[\nb=\\frac{S}{k}=V\\cdot\\frac{n}{k}.\n\\]\nThe stored values of $V$ are rounded to six decimals. Allowing for this rounding, I obtain the intervals in the table.\n\\begin{audittable}{lrrrr}{The printed numbers and the averages over the units that would benefit.}\nData set & $n$ & $k$ & $V=S/n$ (printed as 0.147 and 0.025) & $b=S/k$ \\\\\nHorse Colic & 255 & 62 & 0.146954 & between 0.604405 and 0.604410 \\\\\nStroke Trial & 7264 & 1360 & 0.025186 & between 0.134520 and 0.134526 \\\\\n\\end{audittable}\nNeither interval contains a number that rounds to 0.147 or to 0.025. The identity for $b$ and the two intervals are also proved in Lean, with the stored values and the sample sizes as hypotheses. For an independent least-squares fit of the Stroke model, I find the counts 1360 and 5904 again, with $V=0.025185778762$ and $b=0.134521688917$. The Horse Colic model was not fitted again. Its value of $b$ follows from the stored outputs.", "problem": "The two sentences describe 0.147 and 0.025 as averages over the units that would benefit, that is, over the 62 horses and the 1360 patients whose fitted effect $\\mu_1(x)-\\mu_0(x)$ is positive. The released notebooks compute a different quantity. They multiply each fitted effect by the indicator that it is positive and take the mean over the whole analysis sample: \\texttt{V\\_OPT = ((mu\\_1s - mu\\_0s)* T\\_OPT).mean()}, where \\texttt{T\\_OPT} is the indicator. The units with a fitted effect that is not positive enter this mean as zeros. The stored values are 0.146954 for the 255 horses and 0.025186 for the 7264 patients, and they are the $w_{\\mathrm{TT}}$ utilities of Table 1 (0.14695 and 0.02518). The average over the units that would benefit divides the same sums by 62 and by 1360. It is larger than the printed number by the factors $255/62\\approx4.1$ and $7264/1360\\approx5.3$.", "what_survives": "The utility $V$ as defined in Section 2, Table 1's interpretation as population utilities and regrets, and the theoretical results require no change because of this wording error. If the numbers are kept, then the remarks that follow them (``fairly significant'', and the comparison with the trial's finding on heparin) refer to gains averaged over the whole samples. These are fitted effects of a synthetic model, and my check does not establish clinical or statistical significance." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E009", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of five lines of Appendix E.2.1. The text reads: We trained a logistic regression model of Y on X, T, and the interaction term XT to estimate parameters beta hat, and used the resulting estimate to compute mu sub t of x = sigma(beta hat 0 + beta hat 1 transpose x + beta hat 2 t + beta hat 3 transpose x t). For the fitted model, the AUC was 0.9924, and the accuracy was 0.6824. When computing agent best responses and regret, we take this function mu sub t of x to be given as synthetic mean conditional potential outcomes. A red outline surrounds the words accuracy was 0.6824.", "caption": "Appendix E.2.1 (PDF p. 25). The outline marks the reported accuracy.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E010.png" }, { "alt": "Chart with three rows and a horizontal axis from 0.6 to 1.0, labelled Accuracy: share of the 255 horses counted as correctly classified. The title reads: The printed accuracy, 0.6824, is below every value that correct scoring allows. In the first row, labelled Printed in Appendix E.2.1 (the score the notebook computes), an orange diamond stands at 0.6824 with the note 174 of 255. In the second row, labelled Scored correctly: range implied by the stored outputs of the fit, a blue bar runs from 0.7804 to 0.9843. In the third row, labelled Scored correctly: fit reconstructed from the released coefficients, a blue dot stands at 0.9647 with the note 246 of 255.", "caption": "The accuracy printed in Appendix E.2.1 (orange diamond) beside the correctly scored accuracy of the same fit. The blue bar is the range from $199/255$ to $251/255$ that the score and the AUC stored in the released notebook allow. The blue dot is the value $246/255$ in the reconstruction of the fit from the released coefficients.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E010-accuracy.png" } ], "formal_release": null, "formal_statements": [], "headline": "The accuracy $0.6824$ reported for the outcome model of the Horse Colic data is the share of horses that were predicted to live and did live, $174/255$, because the code compares predictions coded $0$ and $1$ with outcomes coded $-1$ and $1$; the correctly scored accuracy of the same fit is at least $199/255 \\approx 0.7804$, and it is $246/255 \\approx 0.9647$ in a reconstruction of the fit.", "id": "P006-E010", "location": "Appendix E.2.1, sentence “For the fitted model, the AUC was 0.9924, and the accuracy was 0.6824” (PDF p. 25); Horse Colic notebook of the replication files, eighth cell, methods fit_outcome and print_metrics", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Appendix~E.2.1.} For the fitted model, the AUC was 0.9924, and the accuracy was \\hl{0.6824}.\n\n\\emph{Notebook \\texttt{horse\\_colic\\_experiments.ipynb}, method \\texttt{fit\\_outcome} (an excerpt: the lines between those shown and the arguments of the constructor are omitted, and the print statement is broken here after the format string).}\n\\[\n\\begin{aligned}\n&\\texttt{target = self.get\\_target\\_column(df\\_input)}\\\\\n&\\texttt{clf\\_outcome = LogisticRegression(}\\ldots\\texttt{).fit(inputs, target)}\\\\\n&\\texttt{y\\_true = target}\\\\\n&\\texttt{print(\"Accuracy for outcome model: \\%f\"}\\\\\n&\\qquad\\texttt{\\% clf\\_outcome.score(inputs, }\\hl{\\texttt{df\\_input[self.y\\_column]}}\\texttt{))}\n\\end{aligned}\n\\]\n\n\\emph{The same notebook, method \\texttt{print\\_metrics} (an excerpt, with the same line break).}\n\\[\n\\begin{aligned}\n&\\texttt{y\\_true = self.get\\_target\\_column(df\\_input)}\\\\\n&\\texttt{print(\"Accuracy for outcome model: \\%f\"}\\\\\n&\\qquad\\texttt{\\% self.clf\\_outcome.score(inputs, }\\hl{\\texttt{df\\_input[self.y\\_column]}}\\texttt{))}\n\\end{aligned}\n\\]", "corrected": "\\emph{Notebook.} The score is taken against the recoded outcome, \\texttt{target} in \\texttt{fit\\_outcome} and \\texttt{y\\_true} in \\texttt{print\\_metrics}:\n\\[\n\\begin{aligned}\n&\\texttt{clf\\_outcome.score(inputs, }\\hl{\\texttt{target}}\\texttt{)}\\\\\n&\\texttt{self.clf\\_outcome.score(inputs, }\\hl{\\texttt{y\\_true}}\\texttt{)}\n\\end{aligned}\n\\]\n\n\\emph{Appendix~E.2.1.} For the fitted model, the AUC was 0.9924, and the correctly scored accuracy was \\hl{at least $199/255$ (about $0.7804$)}. To obtain its exact accuracy, rescore the original fit on its training sample with the corrected call. The reconstruction of the fit gives $0.9647$; a replacement taken from a reconstructed or newly fitted model should be identified as such.", "evidence": "The evidence is an exact identity, a bound from the outputs stored in the notebook, and a reconstruction of the fit. For a horse with outcome $y \\in \\{-1, 1\\}$ and prediction $h \\in \\{0, 1\\}$, the code tests whether $h = y$, and a correct classification is $h = (y + 1)/2$. There are four cases.\n\\begin{audittable}{llllr}{The four cases of outcome and prediction, and the number of horses in each case in the reconstruction of the fit.}\nOutcome $y$ & Prediction $h$ & Counted by the code & Correctly classified & Horses \\\\\nlived ($1$) & lives ($1$) & yes & yes & 174 \\\\\nlived ($1$) & dies ($0$) & no & no & 4 \\\\\ndied ($-1$) & dies ($0$) & no & yes & 72 \\\\\ndied ($-1$) & lives ($1$) & no & no & 5 \\\\\n\\end{audittable}\nThe table shows that the score of the code is the number of correctly predicted survivors divided by $n$, and that the accuracy adds the number of correctly predicted deaths to the numerator. The sample has $n = 255$ horses, of which 178 lived and 77 died. The notebook stores the score $0.682353$, and $174$ is the only integer $k$ such that $k/255$ rounds to this value. Hence 174 survivors are predicted correctly and 4 are not. The stored AUC, $0.992412$, bounds the number $F$ of deaths that are predicted as survivals. Each of the $4F$ pairs of a misclassified survivor and a misclassified death loses at least half a credit in the AUC, and the total loss over the $178 \\cdot 77 = 13706$ pairs of a survivor and a death is at most $104.01$. This implies that $F \\le 52$. At least 25 deaths are then predicted correctly, and the accuracy of the fit is at least $199/255 \\approx 0.7804$. No such value rounds to $0.6824$.\n\nThe notebook does not store the fitted model or its intercept. I reconstruct the fit from the rounded coefficients that the notebook prints, with the intercept equal to the sum of the coefficients of the two age indicators. The reconstruction reproduces the stored score and the stored AUC to six decimals. Its counts are those in the last column of the table, and its accuracy is $246/255 \\approx 0.9647$. This value belongs to the reconstruction; it is not a rescoring of the original fit.", "problem": "Accuracy is the share of horses whose outcome the fitted classifier predicts correctly, and a correctly predicted death counts as much as a correctly predicted survival. The method \\texttt{get\\_target\\_column} recodes the outcome from $-1$ and $1$ to $0$ and $1$. The classifier is fitted to the recoded outcome, and its predictions take the values $0$ and $1$. Both accuracy statements then compare these predictions with the original outcome, coded $-1$ and $1$. Since a prediction of death is $0$ and a death is coded $-1$ in the original outcome, no death is ever counted as correctly classified. The score that the code prints is the share of horses that were predicted to live and did live. Whatever the model, this share cannot exceed the share of survivors, $178/255 \\approx 0.698$. The printed value $0.6824$ understates the fit of the outcome model to the sample, since the correctly scored accuracy is at least $199/255$. The AUC in the same sentence is computed against the recoded outcome and is not affected. The accuracy is printed and not used again, and no other number in the paper depends on it.", "what_survives": "The classifier is trained on the recoded outcome, and the AUC is computed against it. The accuracy appears only inside two print statements. The correction leaves unchanged the fitted probabilities, the synthetic potential outcomes and every utility, regret and treatment rate for the Horse Colic data, including the Horse Colic entries of Tables~1 and~2 and the figures. The theoretical results and the Stroke Trial calculations do not involve this diagnostic." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E010", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the definition of the Stroke Trial outcome. The text reads: outcome variable Y, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically, Y = -2 times the indicator of death, minus the indicator of recurrent stroke, minus 0.5 times the indicator of pulmonary embolism or intracranial bleeding, minus 0.5 times the indicator of other side effects, plus 2 times the indicator of full recovery at 6 months, plus the indicator of discharge within 14 days. This results in Y in the interval from -4 to 3. A red outline surrounds the last term, the indicator of discharge within 14 days.", "caption": "Appendix E.2.2 (PDF p. 25): the definition of the outcome score. The outline marks the last term, which the released code computes without the 14-day condition.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E011.png" }, { "alt": "Bar chart. The horizontal axis is days from randomization to discharge alive from hospital, from 0 to 60, and the vertical axis is the number of patients discharged alive on that day, from 0 to about 350. The title reads: The code credits 971 discharges after day 14 that the printed score excludes. Blue bars for days 0 to 14 rise from 20 patients on day 0 to between 167 and 268 patients a day from day 3 to day 13, with a peak of 339 on day 14; the legend reads Day 0 to 14: 2,921 patients, credited by the printed score and by the code. A vertical line after day 14 is labelled end of the 14-day window of the printed score. Orange bars for the later days fall from 139 patients on day 15 to fewer than 10 a day after day 36; the legend reads After day 14: 971 patients, credited by the code only. A note at the right says: 52 more patients after day 60 (latest: day 206).", "caption": "Day of discharge of the 3,892 patients of the analyzed sample who were discharged alive, in days from randomization. The released code gives each of them the point for discharge. The printed score gives it to the 2,921 patients discharged on days 0 to 14 (blue) and not to the 971 discharged later (orange). The 52 discharges after day 60 are not drawn.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E011-discharge-days.png" } ], "formal_release": null, "formal_statements": [], "headline": "The released code credits every discharge alive from hospital, while the printed score credits only a discharge within 14 days; 971 of the 7,264 outcomes are one point higher than under the printed score, and the Stroke Trial results in the paper are those of the outcome as coded.", "id": "P006-E011", "location": "Appendix E.2.2, displayed outcome formula, last term (PDF p. 25); Stroke Trial notebook of the replication files, seventh and thirteenth cells; Stroke Trial columns of Table 1 (PDF p. 9) and Table 2 (PDF p. 27), Figure 1 (PDF p. 8) and the Stroke Trial panel of Figure 2 (PDF p. 27)", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Appendix~E.2.2.} For the outcome variable $Y$, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically,\n\\[\n\\begin{aligned}\nY = {} & -2\\,\\mathbb{1}(\\text{death}) - \\mathbb{1}(\\text{recurrent stroke}) - 0.5\\,\\mathbb{1}(\\text{pulmonary embolism or intracranial bleeding})\\\\\n& - 0.5\\,\\mathbb{1}(\\text{other side effects}) + 2\\,\\mathbb{1}(\\text{full recovery at 6 months}) + \\mathbb{1}(\\text{discharge within 14 days}).\n\\end{aligned}\n\\]\nThis results in $Y \\in [-4, 3]$.\n\n\\emph{Notebook \\texttt{stroke\\_trial\\_experiments.ipynb}, thirteenth cell.}\n\\[\n\\texttt{df['discharge'] = }\\hl{\\texttt{(df['DALIVE'] == 'Y')}}\\texttt{.astype(int)}\n\\]", "corrected": "\\emph{Notebook, thirteenth cell (one line of code, broken here after the first condition).}\n\\[\n\\begin{aligned}\n&\\texttt{df['discharge'] = ((df['DALIVE'] == 'Y')}\\\\\n&\\qquad\\hl{\\texttt{\\& df['DALIVED'].between(0, 14, inclusive='both')}}\\texttt{).astype(int)}\n\\end{aligned}\n\\]\nThe cells that follow are then run again. The recomputed values replace the fit statistics of Appendix~E.2.2, the counts and averages for the Stroke Trial data in Section~6, the Stroke Trial columns of Tables~1 and~2, Figure~1 and the Stroke Trial panel of Figure~2. The printed score stays as it is.", "evidence": "The evidence is an exact counterexample in the trial data and an exact refit. The patient on line 30 of the data file \\texttt{IST\\_corrected.csv} passes every filter of the notebook, was discharged alive on day 20, and has none of the other events of the score. For this patient, I have $Y = 1$ under the code and $Y = 0$ under the printed score. For every patient of the sample who was discharged alive, the day of discharge is recorded as an integer from 0 to 206. Each of the 971 late discharges lowers the outcome by exactly one point under the printed score, and the sum of the outcomes falls from 2,765 to 1,794.\n\nI then compare two least-squares fits in exact rational arithmetic, one for the outcome as coded and one with the discharge term restricted to days 0 to 14. The sample, the covariates, the treatment coding and the calculation of the conditional means are the same in both. For the first fit, $\\gamma_{\\mathrm{marg}}$ and the individual and cumulative regrets match the stored notebook values to within $10^{-12}$. Between the two fits, the optimal utility moves from $0.025186$ to $0.024790$, the number of patients with a positive fitted effect of treatment moves from 1,360 to 1,313, and the optimal decision changes for 355 patients. The table sets the Stroke Trial columns of Table~2 beside the recomputed values; the rows other than $w_{\\mathrm{TO}}$ also appear in Table~1. The printed table truncates some of its entries (see \\fid{P006-E018}).\n\\begin{audittable}{llrrr}{Stroke Trial columns of Table~2. For each reward function, the first row gives the entries as printed, and the second row gives the values recomputed with the discharge term restricted to days 0 to 14.}\nReward function & Entry & Utility & Regret & Treat rate \\\\\n$w_{\\mathrm{ATO}}$ & printed & $0.00004$ & $0.0251$ & $0.0001$ \\\\\n & recomputed & $0.00013$ & $0.0247$ & $0.0001$ \\\\\n$w_{\\mathrm{ATT}}$ & printed & $0.00013$ & $0.0250$ & $0.0001$ \\\\\n & recomputed & $0.00013$ & $0.0247$ & $0.0001$ \\\\\n$w_{\\mathrm{TO}}$ & printed & $-0.08278$ & $0.1079$ & $0.6689$ \\\\\n & recomputed & $-0.07492$ & $0.0997$ & $0.6195$ \\\\\n$w_{\\mathrm{TT}}$ & printed & $0.02518$ & $0.000$ & $0.1872$ \\\\\n & recomputed & $0.02479$ & $0.000$ & $0.1808$ \\\\\n$w_{\\mathrm{TT}}$ (no info) & printed & $-0.04888$ & $0.0741$ & $0.4829$ \\\\\n & recomputed & $-0.04912$ & $0.0739$ & $0.4774$ \\\\\n$w_{\\mathrm{TT}}$ (demographic info) & printed & $-0.06392$ & $0.0891$ & $0.5041$ \\\\\n & recomputed & $-0.06287$ & $0.0877$ & $0.4934$ \\\\\n\\end{audittable}\nThe refit changes the discharge term alone and keeps every other coding of the notebook as released. It uses the released notebook and the corrected second version of the trial data, and it does not speak for any earlier version of the code.", "problem": "The last term of the printed score adds one point for a discharge alive from hospital within 14 days of randomization. The seventh cell of the notebook states the score with the same condition. The code sets the discharge indicator from the variable \\texttt{DALIVE} (discharged alive from hospital) alone, and it never reads \\texttt{DALIVED}, the number of days from randomization to that discharge. A patient discharged alive on day 20, or on day 206, receives the point. In the analyzed sample 3,892 patients were discharged alive, 2,921 of them on days 0 to 14 and 971 later. For these 971 patients, of 7,264, the coded outcome is one point above its value under the printed score. The linear model is fitted to this outcome. The synthetic potential outcomes, the Stroke Trial columns of Tables~1 and~2, Figure~1 and the Stroke Trial panel of Figure~2 are all computed from that fit. The numbers published for the Stroke Trial data are therefore those of an experiment whose outcome differs from the one that the appendix defines. The printed score is a coherent definition, and the repair is to make the code follow it. To change the text to ``discharged alive'' would replace the stated 14-day endpoint by a different one.", "what_survives": "The theoretical results, the Horse Colic experiment, and the sample, covariates and treatment allocation of the Stroke Trial experiment do not depend on the discharge term. Every quantity for the Stroke Trial data that is computed from the fitted outcomes has to be recomputed. The changes are small, and the qualitative findings hold in the refit. The optimal utility remains positive, and the regret of $w_{\\mathrm{TT}}$ with full information is exactly zero. The regret with demographic information, $0.0877$, still exceeds the regret with no information, $0.0739$, and regret is still amplified for the individual features Age, Delay, Sleep symptoms and CT done. Some entries under the printed rule do not change (see \\fid{P006-E001}): $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ each still treat one patient of 7,264, the utility of $w_{\\mathrm{ATT}}$ still rounds to $0.00013$, and the last four cumulative regrets of Figure~1 remain zero." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E011", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of the definition of the Stroke Trial outcome. The text reads: outcome variable Y, we apply the negative of the scalarized composite outcome score from Kallus and Zhou (2018). Specifically, Y = -2 times the indicator of death, minus the indicator of recurrent stroke, minus 0.5 times the indicator of pulmonary embolism or intracranial bleeding, minus 0.5 times the indicator of other side effects, plus 2 times the indicator of full recovery at 6 months, plus the indicator of discharge within 14 days. This results in Y in the interval from -4 to 3. A red outline surrounds the third term, -0.5 times the indicator of pulmonary embolism or intracranial bleeding.", "caption": "Appendix E.2.2 (PDF p. 25): the definition of the outcome score. The outline marks the term for pulmonary embolism or intracranial bleeding, which the released code computes as the sum of two indicators.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E012.png" } ], "formal_release": null, "formal_statements": [], "headline": "The released code adds the indicators of pulmonary embolism and of intracranial bleeding where the printed score has the indicator of either event; no patient in the analyzed sample has both events, and no outcome changes.", "id": "P006-E012", "location": "Appendix E.2.2, displayed outcome formula, third term (PDF p. 25); Stroke Trial notebook of the replication files, tenth cell", "pdf_page": 25, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Appendix~E.2.2.}\n\\[\n\\begin{aligned}\nY = {} & -2\\,\\mathbf{1}(\\text{death}) - \\mathbf{1}(\\text{recurrent stroke}) - 0.5\\,\\mathbf{1}(\\text{pulmonary embolism or intracranial bleeding})\\\\\n& - 0.5\\,\\mathbf{1}(\\text{other side effects}) + 2\\,\\mathbf{1}(\\text{full recovery at 6 months}) + \\mathbf{1}(\\text{discharge within 14 days}).\n\\end{aligned}\n\\]\n\n\\emph{Notebook \\texttt{stroke\\_trial\\_experiments.ipynb}, tenth cell.}\n\\[\n\\texttt{df['emb\\_bleeding'] = df['PE14'] }\\hl{\\texttt{+}}\\texttt{ df['H14']}\n\\]", "corrected": "\\emph{Notebook, tenth cell.}\n\\[\n\\texttt{df['emb\\_bleeding'] = }\\hl{\\texttt{((df['PE14'] > 0) | (df['H14'] > 0)).astype(int)}}\n\\]\nThe printed score stays as it is.", "evidence": "The evidence is a recalculation from the trial data with the filters of the notebook. Of the 7,264 patients in the analyzed sample, 44 have \\texttt{PE14} equal to one, 68 have \\texttt{H14} equal to one, and none has both. Replacing the sum by the indicator of the union leaves all 7,264 outcomes exactly unchanged, and the range of the outcome remains $[-4, 3]$.", "problem": "The printed term is the indicator of the union of two events, and it carries one penalty of $0.5$. The code forms the sum of the two indicators \\texttt{PE14} and \\texttt{H14}. A patient with a pulmonary embolism and intracranial bleeding within 14 days would then receive a penalty of $1$. The mismatch has no effect on the paper: no patient in the analyzed sample has both events, and the sum equals the indicator for every patient.", "what_survives": "Every Stroke Trial outcome is unchanged, so no fitted value or result needs to be recomputed for this correction. The code should follow the printed score all the same, since the sum and the union differ on any data in which the two events occur together." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E012", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of Definition 1 (Uniform Rank Preservation). The text reads: A set of score functions w sub k, k from 1 to K, preserves treatment effect ordering uniformly over X if for all j, k in 1 to K, tau superscript j of x is at least tau superscript k of x for all x in X implies that the maximum over pi sub j of E of w sub j of X superscript k, T superscript pi sub j, Y superscript j is at least the maximum over pi sub k of E of w sub k of X superscript k, T superscript pi sub k, Y superscript k. A red outline surrounds X superscript k in the score of agent j.", "caption": "Definition 1, PDF p. 5, right column. The outline marks the covariates of agent $k$ in the score of agent $j$. Definition 2, below it on the page, has the same display.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013.png" }, { "alt": "Crop of the start of a proof. The text reads: Proof. Suppose for agents j and k, Pi sub j and Pi sub k are both unconstrained. For w sub k as defined above. First display: the maximum over pi sub j of E of w sub j of X superscript (j), T superscript pi sub j, Y superscript (k) equals n sub j times E of tau superscript (j) of X superscript (j) times the indicator that it is positive. Second display: the same for agent k, with every superscript (k). A red outline surrounds Y superscript (k) in the first display.", "caption": "Proof of Proposition 4, PDF p. 19, first two displays. The outline marks the outcomes of agent $k$ in the score of agent $j$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-b.png" }, { "alt": "Crop of the start of a proof. The text reads: Proof. By the tower property. A two-line display begins with E of w superscript g sub TT of Y, T superscript pi, X, and equals n times the probability that T sub i equals 1 times a conditional expectation of Y sub i minus mu-hat-zero of X sub i, times g of X sub i. A red outline surrounds the arguments Y, T superscript pi, X.", "caption": "Proof of Theorem 1, PDF p. 19. The outline marks the arguments of the score, written in the order $(\\mathbf Y,\\mathbf T^\\pi,\\mathbf X)$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-c.png" }, { "alt": "Crop of three lines of the theorem and a display. The theorem begins: Theorem 2 (Ranking Desiderata Satisfied). Let P sub X superscript (k) be absolutely continuous with respect to P sub X sub 0, and let g sub k equal one over n sub k times the derivative of P sub X sub 0 with respect to P sub X superscript (k). The display, numbered (2), gives w sub k as the sum over i from 1 to n sub k of Y sub i superscript (k) minus mu-hat-zero of X sub i superscript (k), times T sub i, times g sub k of X superscript (k). A red outline surrounds g sub k of X superscript (k), which has no index i.", "caption": "Restatement of Theorem 2 with equation (2), PDF p. 20. The outline marks the weight $g_k(X^{(k)})$ inside the sum over $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-d.png" }, { "alt": "Crop of part of a proof. The text reads: Let Pi sub j, Pi sub k both be unconstrained. Then two displays give the maximal expected scores of agents j and k as E of tau of X sub 0 times the indicator that it is positive, with superscripts (j) and (k); in the first, the score of agent j is written with Y superscript (k), inside a red outline. Below: Definition 1 is immediately satisfied, since tau superscript (j) of x is at least tau superscript (k) of x for all x in X implies E of tau superscript (k) of X sub 0 times the indicator that it is positive, is at least the same expression with superscript (k). A second red outline surrounds the left expression.", "caption": "Proof of Theorem 2, PDF p. 20. The outlines mark the outcomes of agent $k$ in the score of agent $j$, and the left side of the comparison, which has $\\tau^{(k)}$ where $\\tau^{(j)}$ is meant.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E013-e.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T14", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T14" }, { "id": "P006-T30", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T30" } ], "headline": "In the ranking definitions and their proofs, each agent’s score takes that agent’s own covariates and outcomes, in the order $w(\\mathbf X,\\mathbf T,\\mathbf Y)$; the weight in equation (2) is $g_k(X_i^{(k)})$; and the inequality from which Definition 1 follows has agent $j$ on its left side.", "id": "P006-E013", "location": "Definitions 1 and 2, PDF p. 5, right column; Appendix C, proofs of Proposition 4 and Theorem 1, PDF p. 19; equation (2) and proof of Theorem 2, PDF p. 20", "pdf_page": 5, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T14", "P006-T30" ], "text": { "as_printed": "\\emph{Definitions 1 and 2 (PDF p.~5), right side of each implication.}\n\\[\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^k},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)]\n\\]\n\n\\emph{Proof of Proposition 4 (PDF p.~19), first and last displays.}\n\\[\n\\begin{aligned}\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]&=n_jE[\\tau^{(j)}(X^{(j)})\\,\\mathbf{1}(\\tau^{(j)}(X^{(j)})>0)],\\\\\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]&<\\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})].\n\\end{aligned}\n\\]\n\n\\emph{Proof of Theorem 2 (PDF p.~20), first display.}\n\\[\n\\max_{\\pi_j\\in\\Pi_j}E[w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(k)}})]=E[\\tau^{(j)}(X_0)\\,\\mathbf{1}(\\tau^{(j)}(X_0)>0)],\n\\]\n\n\\emph{Proof of Theorem 1 (PDF p.~19), left side of three displays.} $E[w^g_{\\mathrm{TT}}(\\hl{\\mathbf Y,\\mathbf T^\\pi,\\mathbf X})]$\n\n\\emph{Equation (2) (PDF p.~20).}\n\\[\nw_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})=\\sum_{i=1}^{n_k}(Y_i^{(k)}-\\hat\\mu_0(X_i^{(k)}))T_i^{\\pi_k}\\hl{g_k(X^{(k)})}\\qquad(2)\n\\]\n\n\\emph{Proof of Theorem 2 (PDF p.~20).} Definition 1 is immediately satisfied, since $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x\\in\\mathcal X$ implies $\\hl{E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]}\\ge E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]$.", "corrected": "\\emph{Definition 1 (Uniform Rank Preservation).} A set of score functions $\\{w_k\\}_{k=1}^K$ preserves treatment effect ordering uniformly over $\\mathcal X$ if for all $j,k\\in\\{1,\\ldots,K\\}$,\n\\[\n\\tau^j(x)\\ge\\tau^k(x)\\ \\ \\forall x\\in\\mathcal X\\ \\implies\\ \\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^j},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)].\n\\]\n\n\\emph{Definition 2 (Relative Rank Preservation).} A set of score functions $\\{w_k\\}_{k=1}^K$ preserves treatment effect ordering relative to a reference population $P_{X_0}$ with support $\\mathcal X$ if for all $j,k\\in\\{1,\\ldots,K\\}$,\n\\[\nE[\\tau^j(X_0)]\\ge E[\\tau^k(X_0)]\\ \\implies\\ \\max_{\\pi_j\\in\\Pi_j}E[w_j(\\hl{\\mathbf X^j},\\mathbf T^{\\pi_j},\\mathbf Y^j)]\\ \\ge\\ \\max_{\\pi_k\\in\\Pi_k}E[w_k(\\mathbf X^k,\\mathbf T^{\\pi_k},\\mathbf Y^k)].\n\\]\n\n\\emph{Appendix C (PDF pp.~19--20).} In the three displays for agent $j$, the score is $w_j(\\mathbf X^{(j)},\\mathbf T^{\\pi_j},\\hl{\\mathbf Y^{(j)}})$. In the proof of Theorem 1, the left sides read $E[w^g_{\\mathrm{TT}}(\\hl{\\mathbf X,\\mathbf T^\\pi,\\mathbf Y})]$. Equation (2) reads\n\\[\nw_k(\\mathbf X^{(k)},\\mathbf T^{\\pi_k},\\mathbf Y^{(k)})=\\sum_{i=1}^{n_k}(Y_i^{(k)}-\\hat\\mu_0(X_i^{(k)}))T_i^{\\pi_k}\\hl{g_k(X_i^{(k)})}.\\qquad(2)\n\\]\nIn the proof of Theorem 2: Definition 1 is immediately satisfied, since $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x\\in\\mathcal X$ implies $\\hl{E[\\tau^{(j)}(X_0)\\,\\mathbf{1}(\\tau^{(j)}(X_0)>0)]}\\ge E[\\tau^{(k)}(X_0)\\,\\mathbf{1}(\\tau^{(k)}(X_0)>0)]$.", "evidence": "The evidence is a check of the pages against the paper's own definitions, and a derivation. Appendix C.1 (PDF p.~18) defines $w_k$ on agent $k$'s own three vectors of length $n_k$, and Reward Function 4 (PDF p.~6) has $g(X_i)$ inside the sum and the arguments in the order $(\\mathbf X,\\mathbf T^\\pi,\\mathbf Y)$. With the corrected indices, the proof on PDF p.~20 has $E[w_k]=n_kE[\\tau^{(k)}(X_i^{(k)})\\pi_k(X_i^{(k)})g_k(X_i^{(k)})]=E[\\tau^{(k)}(X_0)\\pi_k(X_0)]$, where the second equality is the change of measure under the condition of Theorem 2 as corrected in \\fid{P006-E005}. Over unconstrained policies the maximum is $E[\\max(\\tau^{(k)}(X_0),0)]$. If $\\tau^{(j)}(x)\\ge\\tau^{(k)}(x)$ for all $x$, then $\\max(\\tau^{(j)}(x),0)\\ge\\max(\\tau^{(k)}(x),0)$ for all $x$. Integrating against $P_{X_0}$, I obtain the inequality with $j$ on the left and $k$ on the right.", "problem": "These are slips of index and of argument. In Definitions 1 and 2 the score of agent $j$ is evaluated on the covariates of agent $k$. In three displays of Appendix C it is evaluated on the outcomes of agent $k$. When $n_j\\ne n_k$, the arguments then have different lengths. In the proof of Theorem 1 the arguments of the score are written in the reverse order. In equation (2) the weight is applied to $X^{(k)}$, without the index $i$ of the sum. In the last passage the two sides of the inequality are identical, and the statement to be derived has $\\tau^{(j)}$ on the left.", "what_survives": "These slips do not change the intended ranking results. The intended formulas are fixed by Appendix C.1 and Reward Function 4, and the proofs go through with the indices corrected under the estimator and absolute-continuity conditions of \\fid{P006-E002} and \\fid{P006-E005}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E013", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of four lines of text and one displayed formula. The text reads: When simulating information asymmetry, we compute empirical estimates of gamma marg over the data. For each individual feature in Figures 1 and 3, let X represent the individual feature, and let U represent the set of all other features. We use the full regression result as mu 0 of (x, u). To calculate mu 0 of x where x represents the individual feature, we take the empirical conditional mean over the dataset. The display sets mu 0 of x equal to a fraction. Its numerator is the sum over i from 1 to n of mu 0 of (X sub i, U sub i) times the indicator of X sub i = x. Its denominator is the indicator of X sub i = x alone, and a red outline surrounds it.", "caption": "Appendix E.4 (PDF p. 26). The outline marks the denominator of the empirical conditional mean, a single indicator with no sum over $i$.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E015.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T22", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T22" } ], "headline": "The empirical conditional mean $\\mu_0(x)$ divides by the number of observations with $X_i = x$, which is $\\sum_{i=1}^n \\mathbf{1}(X_i = x)$; the printed denominator is a single indicator with no sum.", "id": "P006-E015", "location": "Appendix E.4, first displayed formula (PDF p. 26)", "pdf_page": 26, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T22" ], "text": { "as_printed": "To calculate $\\mu_0(x)$ where $x$ represents the individual feature, we take the empirical conditional mean over the dataset:\n\\[\n\\mu_0(x) = \\frac{\\sum_{i=1}^n \\mu_0(X_i, U_i)\\,\\mathbf{1}(X_i = x)}{\\hl{\\mathbf{1}(X_i = x)}}.\n\\]", "corrected": "To calculate $\\mu_0(x)$ where $x$ represents the individual feature, we take the empirical conditional mean over the dataset: for every value $x$ such that $\\sum_{i=1}^n \\mathbf{1}(X_i = x) > 0$,\n\\[\n\\mu_0(x) = \\frac{\\sum_{i=1}^n \\mu_0(X_i, U_i)\\,\\mathbf{1}(X_i = x)}{\\hl{\\sum_{i=1}^n} \\mathbf{1}(X_i = x)}.\n\\]", "evidence": "The evidence is a derivation and a check of the code. Let $m_i = \\mu_0(X_i, U_i)$, and let $N(x) = \\sum_{i=1}^n \\mathbf{1}(X_i = x)$ be the number of observations with $X_i = x$. Suppose that $N(x) > 0$. Conditioned on the event $X = x$, the empirical distribution of the data puts mass $1/N(x)$ on each of these observations. The conditional mean of $m$ is then\n\\[\n\\frac{1}{N(x)} \\sum_{i=1}^n m_i\\,\\mathbf{1}(X_i = x) .\n\\]\nFor two observations with $X_i = x$ and fitted means $1$ and $3$, I obtain $2$ from this formula and $4$ from the printed quotient. Both released notebooks compute the group mean, with the statement \\texttt{grouped = df\\_train.groupby(x\\_obs).mean()}, and the values of $\\gamma_{\\mathrm{marg}}$ and the regrets in the paper were computed with the correct denominator.", "problem": "The denominator is one indicator, and no sum binds its index $i$, so the display is not a well-defined formula. If the indicator is read at an observation with $X_i = x$, the display divides the sum of the fitted means by $1$ and returns their sum instead of their mean. The text calls the quantity an empirical conditional mean, which fixes the intended denominator: the number of observations whose feature equals $x$.", "what_survives": "No number in the paper changes. The second display of Appendix~E.4, which defines $\\gamma_{\\mathrm{marg}}$ as the average of $|\\mu_0(X_i) - \\mu_0(X_i, U_i)|$ over the dataset, stands as printed. It evaluates $\\mu_0$ only at observed values of the feature, where the count is positive." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E015", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of seven lines of the left column. The text reads: Status quo: average treated outcome. We begin by analyzing regret under the current common quality measure that rewards the average treated outcome, as done by the mortality measures in the New York and Philidelphia health departments in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022). Lazzarini et al. (2022) also. A red outline surrounds the words Philidelphia health departments.", "caption": "Section 3, paragraph “Status quo: average treated outcome”, PDF p. 4, left column. The outline marks the phrase that names the second jurisdiction and its issuing body.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E016.png" } ], "formal_release": null, "formal_statements": [], "headline": "The report cards studied by Dranove et al. (2003) were published by the New York Department of Health and, for the second jurisdiction, by the Pennsylvania Health Care Cost Containment Council, not by a Philadelphia health department.", "id": "P006-E016", "location": "Section 3, paragraph “Status quo: average treated outcome”, PDF p. 4, left column", "pdf_page": 4, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "We begin by analyzing regret under the current common quality measure that rewards the average treated outcome, as done by the mortality measures in the New York and \\hl{Philidelphia health departments} in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022).", "corrected": "\\ldots as done by the \\hl{mortality report cards published by the New York Department of Health and the Pennsylvania Health Care Cost Containment Council} in the 1990s analyzed by (Dranove et al., 2003), and in many CMS quality measures (Institute, 2022).", "evidence": "The evidence is a check of the page and of the cited article (Dranove, Kessler, McClellan and Satterthwaite, \\emph{Journal of Political Economy} 111(3), 2003, pp.~555--588), which I read in the copy of the journal version posted by one of its authors. Its abstract (p.~556) names ``New York and Pennsylvania''. Its Section II has ``the N.Y. Department of Health publicly released'' (p.~559) and ``the Pennsylvania Health Care Cost Containment Council published'' (p.~560).", "problem": "The second jurisdiction of the cited study is the state of Pennsylvania, and the body that published its report cards was a council, not a health department. The printed word is also a misspelling of Philadelphia. If only the spelling were corrected, the attribution would still be wrong.", "what_survives": "The sentence is a historical example, and nothing depends on it. The regret results of Section 3 are unaffected." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E016", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of five lines of the right column. The text reads: untreated potential outcomes is bounded. Assumption 2 (Bounded Heteogeneity). The effect of U sub i on the conditional untreated potential outcome is bounded as E of the absolute value of mu-zero of X sub i minus mu-zero of X sub i and U sub i, at most gamma sub marg. Note that mu-zero of X sub i equals E of mu-zero of X sub i and U sub i given X sub i. Thus, the het-. A red outline surrounds the word Heteogeneity.", "caption": "Assumption 2, PDF p. 6, right column. The outline marks the misspelled word in the name of the assumption.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017.png" }, { "alt": "Crop of a bar chart titled gamma sub marg per feature. One orange bar labelled None and twenty blue bars, with heights between about 0.44 and 0.65, carry rotated labels: Conciousness, Visuospatial disorder, Hemianopia, Stroke subtype, Dysphasia, Age, Leg/foot deficit, Brainstem/cerebellar signs, Face deficit, Arm/hand deficit, Other deficit, Atrial fibrillation, Sex, Blood pressure, Delay, Infact visible, CT done, Sleep symptoms, Aspirin prior, Heparin prior. Red outlines surround the labels Conciousness and Infact visible.", "caption": "Figure 1, top panel, PDF p. 8. The outlines mark the feature labels “Conciousness” and “Infact visible”. Figure 2 (PDF p. 27) has the same labels.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-b.png" }, { "alt": "Crop of a centered two-line title in bold: Supplementary Materials for Operationalizing Counterfactual Metrics: Incentives, Ranking, and Information Asymmetry. A red outline surrounds the word Operationalizing.", "caption": "Title of the supplement, PDF p. 13. The outline marks the word at which the title departs from the title of the paper, “On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry”.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-c.png" }, { "alt": "Crop of five lines of a numbered list. Item 1, Auxiliary untreated data, says that the principal has an auxiliary dataset of X prime sub j, T prime sub j, Y prime sub j, and may produce an unbiased estimator from it if: (a) mu prime sub zero of x, equal to E of Y prime sub j of 0 given X prime sub j equals x, is identifiable from this dataset. A set of sufficient conditions for this would be if ignorability was satisfied, and the probability that T prime sub j equals 0 given X sub j equals x is positive for all x in X, and SUTVA was satisfied. A red outline surrounds X sub j, which has no prime.", "caption": "Appendix A.1, item 1(a), PDF p. 13. The outline marks $X_j$ in the positivity condition; the other covariates of the item carry a prime.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-d.png" }, { "alt": "Crop of three lines. The text reads: outcomes. However, medical treatment protocols are also generally heavily codified, with online clinical decision support tools becoming increasingly widely used upt (2022). Still, if the regulatory agency does not have full access to patents' medical records, then information asymmetry will arise. Red outlines surround upt (2022) and the word patents'.", "caption": "Appendix F, PDF p. 28. The outlines mark the citation printed as “upt (2022)” and the word “patents”'.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E017-e.png" } ], "formal_release": null, "formal_statements": [], "headline": "The text, the figure labels and the reference list contain the misspellings and typographical slips listed below; none of them affects a result.", "id": "P006-E017", "location": "Assumption 2, PDF p. 6, right column; Figures 1–3, PDF pp. 8, 27 and 28; reference list, PDF p. 11; title of the supplement and Appendix A.1, PDF p. 13; PDF pp. 17, 23, 26 and 28", "pdf_page": 6, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{itemize}\n\\item PDF p.~6: Assumption 2 (Bounded \\hl{Heteogeneity}).\n\\item Figures 1 and 2 (PDF pp.~8 and 27), feature labels: \\hl{Conciousness}; \\hl{Infact} visible.\n\\item PDF p.~11, reference list: \\hl{Jersey} Neyman.\n\\item PDF p.~13, title of the supplement: Supplementary Materials for \\hl{Operationalizing Counterfactual Metrics}: Incentives, Ranking, and Information Asymmetry\n\\item PDF p.~13, Appendix A.1, item 1(a): $P(T'_j=0\\mid\\hl{X_j}=x)>0\\ \\ \\forall x\\in\\mathcal X$\n\\item PDF p.~17, proof of Proposition 1: We construct \\hl{an} family of distributions for which the regret is unbounded.\n\\item PDF p.~23: bounds from \\hl{senstivity} analysis on expected outcome functions\n\\item PDF p.~26: agent's \\hl{preceived} $\\mu_0(x)$ function\n\\item Figures 2 and 3 (PDF pp.~27 and 28), feature label: \\hl{Abdomcentesis} total protein.\n\\item PDF p.~28: increasingly widely used \\hl{upt (2022)}; access to \\hl{patents'} medical records.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Assumption 2 (Bounded \\hl{Heterogeneity}).\n\\item Feature labels: \\hl{Consciousness}; \\hl{Infarct} visible; \\hl{Abdominocentesis} total protein.\n\\item Reference list: \\hl{Jerzy} Neyman.\n\\item Supplementary Materials for \\hl{On Counterfactual Metrics for Social Welfare}: Incentives, Ranking, and Information Asymmetry\n\\item Appendix A.1, item 1(a): $P(T'_j=0\\mid\\hl{X'_j}=x)>0\\ \\ \\forall x\\in\\mathcal X$\n\\item We construct \\hl{a} family of distributions for which the regret is unbounded.\n\\item bounds from \\hl{sensitivity} analysis on expected outcome functions\n\\item agent's \\hl{perceived} $\\mu_0(x)$ function\n\\item increasingly widely used \\hl{(Wolters Kluwer, 2022)}; access to \\hl{patients'} medical records.\n\\end{itemize}", "evidence": "The evidence is a check of the pages. For the feature labels I compared the documentation of the two data sets. The Stroke variables are the patient's conscious state and whether an infarct is visible. The label ``abdomcentesis'' is printed in the variable list of the Horse Colic data, which the paper follows.", "problem": "Each item has an evident intended reading. Three of them are not misspellings. The supplement is headed with a title that differs from the title of the paper on PDF p.~1. In Appendix A.1, item 1(a), the positivity condition is written with $X_j$, although the item concerns the auxiliary data, whose covariate is $X'_j$ everywhere else in the item. On PDF p.~28 a citation is printed as ``upt (2022)'', which identifies no source. The reference list (PDF p.~9) has one entry of 2022 on UpToDate, published by Wolters Kluwer. The misspelling ``Philidelphia'' on PDF p.~4 is corrected in \\fid{P006-E016}.", "what_survives": "Nothing depends on these slips. No numerical result changes." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E017", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Table 2 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility, regret, and the realized treatment rate. The table has six rows, w ATO, w ATT, w TO, w TT, w TT (no info) and w TT (demographic info), and for each of the Horse Colic and Stroke Trial datasets the columns Utility, Regret and Treat rate. The Horse Colic entries of the last two rows are 0.09761, 0.0493 and 0.6274 in both rows, and a red outline surrounds the two values 0.6274. The Stroke Trial entries are 0.00004, 0.0251, 0.0001 for w ATO; 0.00013, 0.0250, 0.0001 for w ATT; -0.08278, 0.1079, 0.6689 for w TO; 0.02518, 0.000, 0.1872 for w TT; -0.04888, 0.0741, 0.4829 for w TT (no info); and -0.06392, 0.0891, 0.5041 for w TT (demographic info).", "caption": "Table 2 (PDF p. 27). The outline marks the Horse Colic treatment rates of the last two rows, printed as 0.6274.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E018.png" }, { "alt": "Crop of Table 1 with its caption: Utility and regret comparisons for different reward functions. For each reward function w, we report utility, regret, and the realized treatment rate. The table has five rows, w ATO, w ATT, w TT, w TT (no info) and w TT (demographic info), with the same columns as Table 2. The Horse Colic entries of the last two rows are 0.09761, 0.0493 and 0.6275 in both rows, and a red outline surrounds the two values 0.6275. The Stroke Trial entries of the five rows are the same as in Table 2.", "caption": "Table 1 (PDF p. 9). The outline marks the same two treatment rates, printed as 0.6275.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-E018-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "For the Horse Colic data, Table 2 prints the treatment rate of the two rows with information asymmetry as $0.6274$ where Table 1 prints $0.6275$ for the same rate, $160/255$; several Stroke Trial entries and the two fit statistics of Appendix E.2.2 are truncated where neighboring entries are rounded.", "id": "P006-E018", "location": "Table 2, Horse Colic “Treat rate” in the rows $w_{\\mathrm{TT}}$ (no info) and $w_{\\mathrm{TT}}$ (demographic info) (PDF p. 27), against the same cells of Table 1 (PDF p. 9); Stroke Trial entries of Tables 1 and 2, and the fit statistics of Appendix E.2.2 (PDF p. 25)", "pdf_page": 27, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{Table~1 (PDF p.~9), Horse Colic dataset, Treat rate.} $w_{\\mathrm{TT}}$ (no info): 0.6275. $w_{\\mathrm{TT}}$ (demographic info): 0.6275.\n\n\\emph{Table~2 (PDF p.~27), the same cells.} $w_{\\mathrm{TT}}$ (no info): \\hl{0.6274}. $w_{\\mathrm{TT}}$ (demographic info): \\hl{0.6274}.\n\n\\emph{Tables~1 and~2, Stroke Trial dataset.} $w_{\\mathrm{TT}}$, Utility: 0.0251\\hl{8}. $w_{\\mathrm{TT}}$ (no info), Utility: $-0.0488\\hl{8}$. $w_{\\mathrm{ATT}}$, Regret: 0.025\\hl{0}.\n\n\\emph{Table~2, Stroke Trial dataset.} $w_{\\mathrm{TO}}$, Utility: $-0.0827\\hl{8}$; Regret: 0.10\\hl{79}.\n\n\\emph{Appendix~E.2.2 (PDF p.~25).} The fitted RMSE was 1.3\\hl{4} and the $R^2$ was 0.2\\hl{6}.", "corrected": "\\emph{Table~2, Horse Colic dataset, Treat rate.} $w_{\\mathrm{TT}}$ (no info): \\hl{0.6275}. $w_{\\mathrm{TT}}$ (demographic info): \\hl{0.6275}.\n\nIf every entry is to be rounded to nearest, the Stroke Trial entries become $0.0251\\hl{9}$, $-0.0488\\hl{9}$ and $0.025\\hl{1}$ in both tables and $-0.0827\\hl{7}$ and $0.10\\hl{80}$ in Table~2, and the sentence of Appendix~E.2.2 reads: The fitted RMSE was 1.3\\hl{5} and the $R^2$ was 0.2\\hl{7}.", "evidence": "The evidence is a recalculation from the outputs stored in the notebooks and an exact refit of the Stroke Trial model. The notebook for the Horse Colic data prints both rates as $0.627451$, and 160 is the only count out of 255 that agrees with this output. For the Stroke Trial data the computed values are $0.0251858$, $-0.0488860$ and $0.0250547$ for the three entries common to both tables, $-0.0827749$ and $0.1079606$ for the $w_{\\mathrm{TO}}$ row, and $1.346663$ and $0.269755$ for the RMSE and the $R^2$. All the printed entries are the truncated values except $-0.08278$, which is neither the rounded nor the truncated value of $-0.0827749$. A second rounding of the six-decimal output of the notebook, $-0.082775$, would produce it.", "problem": "Table~2 repeats the rows of Table~1 and adds the row $w_{\\mathrm{TO}}$. The two treatment rates are the same quantities in both tables. Each rate is $160/255 = 0.62745\\ldots$, which rounds to $0.6275$, and Table~2 shows the truncated value. The other entries listed above also differ from the nearest rounding at the printed precision, while most entries of the two tables are rounded to nearest. The paper states no rounding rule, so for these entries the defect is an inconsistency of presentation and not a wrong value.", "what_survives": "No sign, ordering or conclusion depends on these digits, and the calculations and figures use the unrounded values. The Stroke Trial values here are those of the outcome as coded in the released notebook, which is the subject of \\fid{P006-E011}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-E018", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of five lines of the left column. The text reads: pal were to estimate mu-hat-zero of x from the agent's untreated units with T sub i equal to 0. We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to Theorem 5. Most importantly, our key result is that even without confounding, informa-. A red outline surrounds the words Theorem 5.", "caption": "Section 5, paragraph “Information asymmetry and confounding”, PDF p. 7, left column. The outline marks the theorem number.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N001.png" } ], "formal_release": null, "formal_statements": [], "headline": "The bound to which the appendix result is compared is that of Theorem 3, not Theorem 5.", "id": "P006-N001", "location": "Section 5, paragraph “Information asymmetry and confounding”, PDF p. 7, left column", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to \\hl{Theorem 5}.", "corrected": "We discuss this case in detail in the Appendix, and show that a stronger assumption yields a similar bound to \\hl{Theorem 3}.", "evidence": "The evidence is a check of the pages. The introduction to Appendix D (PDF p.~20) has the reference right: ``we can get similar regret bounds to Theorem 3''.", "problem": "Theorem 5 is the appendix result that the sentence announces: under the stronger Assumption 4, the regret is at most $2\\gamma_{\\max}$ (PDF p.~23). The bound with which it is compared is that of Theorem 3, $R(\\pi^{w_{\\mathrm{TT}}})\\le2\\gamma_{\\mathrm{marg}}$, stated on the same page as the sentence.", "what_survives": "Nothing depends on the cross-reference." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N001", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of four lines of the right column. The text reads: setup by comparing treating with high doses of heparin and aspirin (T = 1) with aspirin alone (T = 0). This leaves n = 7264 patients and a treatment rate of 0.33. The 20 covariates include each patient's age. A red outline surrounds the words n = 7264 patients.", "caption": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 7, right column. The outline marks the size of the analysis sample.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N003.png" } ], "formal_release": null, "formal_statements": [], "headline": "The Stroke sample of 7264 patients includes 35 whose status at six months is missing; the released notebook keeps them and scores them as neither dead nor recovered, and without them the sample has 7229 patients.", "id": "P006-N003", "location": "Section 6, paragraph “International Stroke Trial dataset”, PDF p. 7, right column; replication notebook stroke_trial_experiments.ipynb, the cell headed “Filter missing outcomes”", "pdf_page": 7, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\emph{PDF p.~7.} This leaves \\hl{$n=7264$ patients} and a treatment rate of 0.33.\n\n\\emph{Replication notebook.} Under the comment \\texttt{\\# Filter missing outcomes}, the filter is \\texttt{df = df[df['OCCODE'] \\hl{< 5}]}. Later cells set \\texttt{df['death'] = (df['OCCODE'] == 1).astype(int)} and \\texttt{df['recovery'] = (df['OCCODE'] == 4).astype(int)}.", "corrected": "\\emph{Replication notebook.} \\texttt{df = df[df['OCCODE']\\hl{.isin([1, 2, 3, 4])}]}\n\n\\emph{PDF p.~7.} This leaves \\hl{$n=7229$ patients} and a treatment rate of 0.33, where patients whose status at six months is missing are excluded; the fitted Stroke model, its tables and its figures are computed again on this sample.", "evidence": "The evidence is a recalculation from the replication data, in exact arithmetic. The data file has 19435 rows, with \\texttt{OCCODE} equal to 0 in 97, to 9 in 53, and to 1, 2, 3 or 4 in the others. When the filter is restricted to the codes 1 to 4, exactly the 35 patients are removed, of whom 11 were treated. The least-squares model was then fitted again with no other change.\n\\begin{audittable}{lrr}{Stroke results with the released filter and with the 35 patients removed.}\nQuantity & As released & Codes 1 to 4 only \\\\\nPatients in the analysis sample & 7264 & 7229 \\\\\nPatients with a positive fitted effect & 1360 & 1390 \\\\\nUtility under $w_{\\mathrm{TT}}$ & 0.025186 & 0.026224 \\\\\nRegret under $w_{\\mathrm{TT}}$ & 0 & 0 \\\\\nRegret with no information & 0.074072 & 0.072990 \\\\\nRegret with demographic information & 0.089102 & 0.087954 \\\\\n\\end{audittable}\nThe recalculation does not show what the missing outcomes were, and the direction of any bias is not established.", "problem": "The article that documents the trial database defines the values 0 and 9 of \\texttt{OCCODE} as missing status at six months. The filter removes the code 9 and keeps the code 0. Of the 7264 patients analysed, 35 have the code 0, and their six-month fields for death, dependency and recovery are blank or unknown. The two equality tests then score all 35 as neither dead nor recovered, and neither the paper nor the notebook states a rule for missing outcomes. The printed count 7264 is the size of the sample that the notebook analyses. The defect is in the filter, which does not do what its comment says.", "what_survives": "The theoretical results and the Horse Colic experiment are unaffected. Without the 35 patients the Stroke entries of Tables 1 and 2, Figure 1, the Stroke panel of Figure 2 and the fit statistics of Appendix E.2.2 change slightly or not at all, and the conclusions drawn from them stand: the utility under $w_{\\mathrm{TT}}$ is positive, the regret with demographic information exceeds the regret with no information, and the order of the features is unchanged. The observed treatment rate still rounds to 0.33. The table changes the filter only. A second correction to the Stroke outcome is the subject of \\fid{P006-E011}." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N003", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of ten lines of the left column, beginning 'We also study a more continuous spectrum of information asymmetry'. A red box marks the words 'ascending order of feature importance'. Four lines below, not marked, the text reads 'building up starting from the most important feature'.", "caption": "Section 6, PDF p. 8, left column. The marked phrase gives the order in which the features are sorted. The last sentence in the crop says that the subsets are built up from the most important feature.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N004.png" } ], "formal_release": null, "formal_statements": [], "headline": "“Ascending order of feature importance” means ascending order of the individual $\\gamma_{\\mathrm{marg}}$, which puts the most important feature first.", "id": "P006-N004", "location": "Section 6, PDF p. 8, left column, paragraph beginning “We also study a more continuous spectrum”", "pdf_page": 8, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "Relating this to Section 5, we first sort the features in \\hl{ascending order of feature importance}, as measured by estimating $\\gamma_{\\mathrm{marg}}$ when the principal knows only the individual feature. Then, Figure 1 shows regret when the principal knows increasingly large feature subsets, building up starting from the most ``important'' feature.", "corrected": "Relating this to Section 5, we first sort the features in \\hl{ascending order of individual $\\gamma_{\\mathrm{marg}}$ (from most to least important)}, as measured by estimating $\\gamma_{\\mathrm{marg}}$ when the principal knows only the individual feature.", "intended_meaning": "The features are sorted by their individual values of $\\gamma_{\\mathrm{marg}}$, from the smallest to the largest. Assumption 2 defines $\\gamma_{\\mathrm{marg}}$ as a bound on $E[|\\mu_0(X_i)-\\mu_0(X_i,U_i)|]$. A feature with a small $\\gamma_{\\mathrm{marg}}$ leaves little of the untreated outcome unexplained when the principal knows that feature alone, and the feature with the smallest value is the most important one. The order is ascending in $\\gamma_{\\mathrm{marg}}$ and descending in importance. The next sentence and the caption of Figure 1 (``accumulates features from the left (most important)'') state this order, and the replication files sort the values of $\\gamma_{\\mathrm{marg}}$ upward for both datasets. Figure 1 is unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N004", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of item 2, 'Untreated patients in the treatment population', with its conditions (a) No information asymmetry and (b) Positivity. Two red boxes mark the end of condition (b): 'and Y i is observed for a nonzero proportion of untreated units for all x.'", "caption": "Appendix A.1, item 2, PDF p. 13. The marked clause is the only condition placed on which untreated outcomes are observed.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N005.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T23", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T23" }, { "id": "P006-T24", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T24" } ], "headline": "The conditions listed for identifying $\\mu_0(x)$ from a fraction of the untreated outcomes are not sufficient; adding mean representativeness of the observed untreated outcomes at each $x$ makes them sufficient.", "id": "P006-N005", "location": "Appendix A.1, item 2 (“Untreated patients in the treatment population”), conditions (a) and (b), PDF p. 13", "pdf_page": 13, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T23", "P006-T24" ], "text": { "as_printed": "2. \\textbf{Untreated patients in the treatment population.} In the main paper, we have assumed that the principal only observed outcomes $Y_i$ for treated units with $T_i^\\pi=1$. If the principal also observed a fraction of outcomes for untreated units, with $T_i^\\pi=0$, then these may also be incorporated into the estimate for $\\hat\\mu_0(x)$. Sufficient conditions for identifying $\\mu_0(x)$ from a dataset of the agent's untreated units are:\n\n(a) No information asymmetry: the agent's treatment policy $\\pi$ only depends on $X_i$. Under this assumption, we have no confounding: $\\{Y_i(0),Y_i(1)\\}\\perp\\!\\!\\!\\perp T_i^\\pi\\mid X_i$.\n\n(b) Positivity: $P(T_i^\\pi=0\\mid X_i=x)>0$ for all $x\\in\\mathcal{X}$, \\hl{and $Y_i$ is observed for a nonzero proportion of untreated units for all $x$.}", "corrected": "Sufficient conditions for identifying $\\mu_0(x)$ from a dataset of the agent's untreated units are:\n\n(a) No information asymmetry: the agent's treatment policy $\\pi$ only depends on $X_i$. Under this assumption, we have no confounding: $\\{Y_i(0),Y_i(1)\\}\\perp\\!\\!\\!\\perp T_i^\\pi\\mid X_i$.\n\n(b) Positivity: $P(T_i^\\pi=0\\mid X_i=x)>0$ for all $x\\in\\mathcal{X}$, and $Y_i$ is observed for a nonzero proportion of untreated units for all $x$.\n\n\\hl{(c) Representative observation:} with $R_i=1$ when $Y_i$ is observed,\n\\[\n\\hl{E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]}\\quad\\text{for all }x\\in\\mathcal{X}.\n\\]\n\nUnder (a), (b), (c) and consistency, $\\mu_0(x)=E[Y_i\\mid X_i=x,T_i^\\pi=0,R_i=1]$. Condition (c) holds when $Y_i(0)$ is independent of $R_i$ given $X_i$ and $T_i^\\pi=0$.", "evidence": "The evidence is an exact counterexample and a derivation of the corrected statement. Let $X=0$ and $Y(1)=1$ for every unit, and let treatment be assigned with probability $1/2$ independently of the potential outcomes. Every treated outcome is observed. Let $R=1$ when the outcome of a unit is observed. I consider two distributions of the untreated outcome $Y(0)\\in\\{0,1\\}$ and of $R$ among the untreated units.\n\\begin{audittable}{lrr}{Two distributions that satisfy (a) and (b) and give the same observed data.}\nQuantity & First distribution & Second distribution \\\\\n$P(Y(0)=1)$ & $1/2$ & $3/4$ \\\\\n$P(R=1\\mid Y(0)=0,\\,T=0)$ & $1/4$ & $1/2$ \\\\\n$P(R=1\\mid Y(0)=1,\\,T=0)$ & $3/4$ & $1/2$ \\\\\n$P(R=1\\mid T=0)$ & $1/2$ & $1/2$ \\\\\nMean of the observed untreated outcomes & $3/4$ & $3/4$ \\\\\n$\\mu_0(0)$ & $1/2$ & $3/4$\n\\end{audittable}\nIn both distributions the observed data have the same law: an untreated unit with a missing outcome has probability $1/4$, an untreated unit with observed outcome $0$ has probability $1/16$, an untreated unit with observed outcome $1$ has probability $3/16$, and a treated unit, whose outcome is $1$, has probability $1/2$. Since the two values of $\\mu_0(0)$ differ, $\\mu_0(0)$ is not a function of the law of the observed data.\n\nFor the corrected statement, suppose in addition that $E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]$. By consistency, this condition and (a), I have\n\\[\n\\begin{aligned}\nE[Y_i\\mid X_i=x,T_i^\\pi=0,R_i=1]&=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0,R_i=1]\\\\\n&=E[Y_i(0)\\mid X_i=x,T_i^\\pi=0]=E[Y_i(0)\\mid X_i=x]=\\mu_0(x),\n\\end{aligned}\n\\]\nand the conditioning event has positive probability by (b).", "problem": "Item 2 supposes that the principal observes the outcomes of only a fraction of the untreated units. Among the listed conditions, (a) concerns how treatment is assigned, and (b) requires only that some untreated outcomes are observed at each $x$. Neither condition says which untreated outcomes are observed. If the probability that an untreated outcome is observed depends on its value, then the mean of the observed untreated outcomes at $x$ can differ from the mean of all untreated outcomes there. In that case (a) and (b) hold and $\\mu_0(x)=E[Y_i(0)\\mid X_i=x]$ is not identified.", "what_survives": "The correction concerns one way of obtaining the estimate $\\hat\\mu_0$. It does not affect the results that take an unbiased estimate as given, the auxiliary data of item 1 and Appendix D.2, or the analysis in Appendix D.3, which uses the mean over all untreated units. Appendix E.3 states that the experiments do not simulate the estimation of $\\hat\\mu_0$ from a subset of the data, and no reported number changes. The added condition is an assumption about how outcomes come to be observed, and a data source in which observation depends on the outcome can fail to satisfy it. The first paragraph of Appendix D.3 needs the same condition when only a fraction of the untreated outcomes is observed." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N005", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [ { "alt": "Crop of four lines of the proof of Proposition 5 around the display 'E of w TT equals n, open parenthesis, E of pi times (mu 1 minus mu 0)'. A red box marks the letter n and the opening parenthesis that follows it.", "caption": "Appendix D.2, proof of Proposition 5, PDF p. 22. The marked parenthesis after $n$ has no closing partner.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N006.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T25", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T25" } ], "headline": "The display for $E[w_{\\mathrm{TT}}]$ in the proof of Proposition 5 has an opening parenthesis after $n$ that is never closed.", "historical_evidence_type": "lean_proof", "id": "P006-N006", "location": "Appendix D.2, proof of Proposition 5, display before “The resulting policy $\\pi^{w_{\\mathrm{TT}}}$ that maximizes this is”, PDF p. 22", "pdf_page": 22, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P006-T25" ], "text": { "as_printed": "Then by Lemma 2, and assuming $\\hat\\mu_0(x)$ is unbiased, we have\n\\[\nE[w_{\\mathrm{TT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=n\\hl{(}E[\\pi(X_i,U_i)(\\mu_1(X_i,U_i)-\\mu_0(X_i))].\n\\]\nThe resulting policy $\\pi^{w_{\\mathrm{TT}}}$ that maximizes this is", "corrected": "\\[\nE[w_{\\mathrm{TT}}(\\mathbf{X},\\mathbf{T}^\\pi,\\mathbf{Y})]=\\hl{nE}[\\pi(X_i,U_i)(\\mu_1(X_i,U_i)-\\mu_0(X_i))].\n\\]", "evidence": "I checked the unmatched parenthesis on the page. The expected-reward identity with the parenthesis removed is also proved in the Lean statements of \\fid{P006-T25}.", "problem": "The parenthesis after $n$ is a misprint; the expected reward is $n$ times the displayed expectation.", "what_survives": "Proposition 5 and its proof are unchanged. The displayed policy is one of several maximizers, since the agent is indifferent at $u=1$; Section 2 defines $\\pi^w$ as a member of the set of maximizers, and the printed regret $\\alpha-\\beta/2$ is the regret of the displayed policy." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N006", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of five lines of text. The text reads: Figure 3 is the equivalent of Figure 1 for the Horse Colic dataset. For the Horse Colic dataset, the regret drops to close to zero if the principal knows only a few features. This is likely due to the fact that there is little variation in other feature values in the dataset conditioned packed cell volume and pulse, exacerbated by the fact that the dataset is so small. The value of gamma marg for these two features combined is small compared to the individual gamma marg values in Figure 3 (left), at 0.035. A red outline surrounds the value 0.035.", "caption": "Appendix E.5, paragraph below Figure 2 (PDF p. 27). The outline marks the reported value 0.035.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N009.png" } ], "formal_release": null, "formal_statements": [], "headline": "For the fit recorded in the released notebook, the joint value of $\\gamma_{\\mathrm{marg}}$ for packed cell volume and pulse in the Horse Colic data is $0.028$ and not the printed $0.035$.", "id": "P006-N009", "location": "Appendix E.5, paragraph below Figure 2, last sentence (PDF p. 27)", "pdf_page": 27, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "The value of $\\gamma_{\\mathrm{marg}}$ for these two features combined is small compared to the individual $\\gamma_{\\mathrm{marg}}$ values in Figure 3 (left), at \\hl{0.035}.", "corrected": "The value of $\\gamma_{\\mathrm{marg}}$ for these two features combined is small compared to the individual $\\gamma_{\\mathrm{marg}}$ values in Figure 3 (left), at \\hl{0.028}.\n\nThe release can make this value explicit with one more cell, which evaluates\n\\[\n\\texttt{mean(abs(mu\\_0s\\_marginal - mu\\_0s))}\n\\]\nfor the pair of features \\texttt{x\\_obs = ['X\\_18', 'X\\_4']}.", "evidence": "The evidence is a derivation of exact bounds. The notebook prints the coefficients of the fit to eight decimals and does not print the intercept. The mean of the fitted untreated outcome over the 232 adult horses, which the notebook prints as $0.542912$, confines the intercept to the interval $(3.466, 3.472)$. Carrying the rounding intervals of the coefficients and this interval for the intercept through the calculation in rational arithmetic, I find that the joint $\\gamma_{\\mathrm{marg}}$ lies in $[0.028006, 0.028055]$ for every fit consistent with the printed output. The two features define 211 groups, and 177 of the 255 horses are alone in their group.", "problem": "The sentence reports the empirical $\\gamma_{\\mathrm{marg}}$ of Appendix~E.4 when the principal observes packed cell volume and pulse together. This is the average over the 255 horses of $|\\mu_0(X_i) - \\mu_0(X_i, U_i)|$, where $\\mu_0(X_i)$ is the mean of the fitted untreated outcome over the horses with the same two values. For the fit recorded in the released notebook this average rounds to $0.028$ to three decimals, and $0.035$ is not its value at the printed precision. The notebook computes $\\gamma_{\\mathrm{marg}}$ for single features and regret for cumulative sets of features. It contains no calculation of the joint value, and the origin of the number $0.035$ is not established.", "what_survives": "The comparison that the sentence makes stands: $0.028$ is small beside the values for the single features, $0.393$ for packed cell volume and $0.417$ for pulse. No calculation uses the printed number. The tables, Figure~3 and the regret after these two features, about $0.0008$ in the notebook, are unchanged." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N009", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "historical_lean_attribution", "exhibits": [ { "alt": "Crop of six lines of text. A red box marks the words: the bound in Assumption 4 implies that Assumption 2 is satisfied with gamma marg less than or equal to gamma max.", "caption": "Appendix D.3, PDF p. 23: the implication from Assumption 4 to Assumption 2.", "url": "https://pmaronow.github.io/llm-audit/works/P006/exhibits/P006-N011.png" } ], "formal_release": null, "formal_statements": [ { "id": "P006-T17", "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T17" } ], "headline": "The statement that Assumption 4 implies Assumption 2 with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$ holds when the private type has a conditional distribution given the covariates, as it does when it takes finitely many values or values in a Euclidean space; on general measurable spaces the implication can fail.", "historical_evidence_type": "lean_proof", "id": "P006-N011", "location": "Appendix D.3, PDF p. 23, paragraph after Assumption 4; Assumption 2, PDF p. 6", "pdf_page": 23, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P006-T17" ], "text": { "as_printed": "Appendix D.3, PDF p.~23: ``Furthermore, \\hl{the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$}.''\n\nAssumption 4: ``Define the maximum difference in the untreated potential outcome for a given $x,u$ to be $\\Delta(x,u)=\\max_{\\tilde u\\in\\mathcal U}|\\mu_0(x,u)-\\mu_0(x,\\tilde u)|$. The average difference is bounded as: $E[\\Delta(X,U)]\\le\\gamma_{\\max}$.''\n\nAssumption 2, PDF p.~6: ``The effect of $U_i$ on the conditional untreated potential outcome is bounded as $E[|\\mu_0(X_i)-\\mu_0(X_i,U_i)|]\\le\\gamma_{\\mathrm{marg}}$.'' ``Note that $\\mu_0(X_i)=E[\\mu_0(X_i,U_i)\\mid X_i]$.''", "corrected": "Appendix D.3: ``Furthermore, \\hl{when $U$ has a conditional distribution given $X$, so that $\\mu_0(x)$ is the average of $\\mu_0(x,u)$ over it,} the bound in Assumption 4 implies that Assumption 2 is satisfied with $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$.''", "intended_meaning": "If the private type $U$ has a conditional distribution given $X$, then for almost every $x$, $\\mu_0(x)$ is the average of $\\mu_0(x,v)$ over that distribution, so $|\\mu_0(x)-\\mu_0(x,u)|\\le\\Delta(x,u)$ for every $u$. With the usual integrability, taking expectations gives $E[|\\mu_0(X)-\\mu_0(X,U)|]\\le E[\\Delta(X,U)]\\le\\gamma_{\\max}$, so Assumption 2 holds with a constant $\\gamma_{\\mathrm{marg}}\\le\\gamma_{\\max}$. A conditional distribution exists when $U$ takes finitely or countably many values and, more generally, when it takes values in a standard Borel space such as a Euclidean space, which covers the settings the paper describes. The paper does not state this condition. On a general measurable space, where $\\mu_0(X)$ is defined only as a conditional expectation, the implication can fail: in an example whose type space admits no such conditional distribution, $E[|\\mu_0(X)-\\mu_0(X,U)|]=3/4$ while $E[\\Delta(X,U)]=1/2$. The example concerns only this comparison. It does not bear on models in which $U$ has a conditional distribution given $X$, and the finite-type calculation of \\fid{P006-E014} is not affected." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-N011", "work": { "arxiv_id": "2305.14595", "doi": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "url": "https://pmaronow.github.io/llm-audit/works/P006/" }, "work_id": "P006" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P006.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "Exact witness for the printed assertions, not a positive proof of either printed proposition; the credited helper rows justify its probability, SUTVA, ATT and welfare interpretations.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T1_average_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L337", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/P006_CounterfactualMetrics.lean:2437 gives exactly T1(c): the treated population mean is 1 and the unconditional two-unit reward is 3/4. In this specified law the treatment probability is 1/2 and only the type with outcome 1 is treated, so finiteConditionalAverage is precisely the printed conditional mean. The finite ratio introduces neither a zero-denominator convention nor a restriction on this counterexample.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T1_conditional_mean_mismatch", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2096", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "The exact Bernoulli probability helper is not separately invoked; the fixed finite model is explicit in the definitions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.coin_is_probability", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L291", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "Treatment counts are already present in the exact printed reward definition and the exported witness theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.treatedCount_two", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L298", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "The realized reward expansion is already incorporated in the specialized witness; SUTVA transport uses the separately credited invariance row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.wATO_two", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L305", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "The specialized expected-reward polynomial is used directly instead of the more general two-unit expansion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.twoATO_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L313", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "The optimizer row supplies feasibility separately; no restriction of the stochastic policy square is hidden here.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.rewardE001_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L326", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "This auxiliary equality transfers the concrete zero-baseline ATO counterexample to ATT.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.wATT_zero", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2088", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "TT follows by the allowed specialization g=1; n=0 is also covered and harmless.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.realized_rewards_SUTVA", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2498", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "conditional_on": [], "credit_reason": "Harmless algebraic generalization; outside the probability range the left side is a signed weighted sum rather than an expectation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.binaryWelfare_SUTVA", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2537", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f1b26a79b21d50f33689b11ccaef381d86caaa31bb24e1c4c81eed486bc84423", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T1", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T1" }, "html": { "body": "

PDF p. 15 (Proposition 6 and proof, Appendix B.3); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3, after Reward Functions 1 and 2). The pages are unnumbered. The specified two-unit experiment establishes the canonical counterexample to the paper's printed Propositions 6 and 7 and their first proof displays, including the unique finite-sample optimum, reward polynomial, and welfare gaps.

", "headline": "Proposition 6 (ATO Best Response) and Proposition 7 (ATT Best Response), as printed, with the first display of each proof and the same best-response rules in Section 3", "location": "PDF p. 15 (Proposition 6 and proof, Appendix B.3); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3, after Reward Functions 1 and 2). The pages are unnumbered." }, "id": "P006-T1", "text": { "body": "PDF p. 15 (Proposition 6 and proof, Appendix B.3); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3, after Reward Functions 1 and 2). The pages are unnumbered. The specified two-unit experiment establishes the canonical counterexample to the paper's printed Propositions 6 and 7 and their first proof displays, including the unique finite-sample optimum, reward polynomial, and welfare gaps.", "headline": "Proposition 6 (ATO Best Response) and Proposition 7 (ATT Best Response), as printed, with the first display of each proof and the same best-response rules in Section 3", "location": "PDF p. 15 (Proposition 6 and proof, Appendix B.3); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3, after Reward Functions 1 and 2). The pages are unnumbered." }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T1" }, { "declarations": [ { "claim_revision_id": "catalog:b71c9f48d7ab7beb86fd84c94dd228ce9112bfd25e57d3ef5cb409a8ebcde134", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P006/Wave1/Gap/FidP006.lean:99 states the empty and nonempty probabilities, the corrected unconditional objectives, the reward means conditional on a nonempty sample, and the corresponding best-response equivalences. Countable covariate support represents the target's discrete distribution, and evaluated-estimator integrability supplies the ordinary conditional expectation required by the ATT premise. The ATO implication has no estimator assumption. The separate single-unit identity E[m(X_i) | T_i=1]=A/q is not asserted by the component assigned to this row.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T2_corrected_average_split", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/FidP006.lean#L99", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:b71c9f48d7ab7beb86fd84c94dd228ce9112bfd25e57d3ef5cb409a8ebcde134", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/Wave1/Gap/GP006B1.lean:229 covers the finite-support frontier and conditional-average optimization in T2(iii)-(iv), whose claim expressly assumes finite positive-mass support and ordered means. Its exact frontier at every interval together with an attained optimal threshold gives the claimed scalar optimization over q; zero-based indices and prefix sums change no cases. The n=1 reduction is immediate from the definition of emptyFactor at P006/P006_CounterfactualMetrics.lean:82.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T2_finite_threshold_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B1.lean#L226", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:b71c9f48d7ab7beb86fd84c94dd228ce9112bfd25e57d3ef5cb409a8ebcde134", "conditional_on": [], "credit_reason": "The component is the one-unit multiplier identity, not the complete general expectation theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T2_one_unit_factor", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L701", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:b71c9f48d7ab7beb86fd84c94dd228ce9112bfd25e57d3ef5cb409a8ebcde134", "conditional_on": [], "credit_reason": "The supplied maximizing index forces nonempty support; strictly positive masses exactly fit the support used in target T2(iii)-(iv).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T2_population_conditional_rule", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2284", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b71c9f48d7ab7beb86fd84c94dd228ce9112bfd25e57d3ef5cb409a8ebcde134", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T2", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T2" }, "html": { "body": "

PDF pp. 15-16 (Propositions 6 and 7); PDF pp. 4-5 (Section 3). The zero-on-empty ATO and conditionally unbiased ATT rewards have exact expectation h_n(q) A, empty probability (1-q)^n, and conditional means A/q when q>0. Their best responses maximize this finite-sample objective. On finite support the fixed-mass frontier is attained by a fractional threshold, and maximizing it over q gives an attained threshold optimum. Such an optimum can treat a nonmaximal score. The printed positive-maximizer rule optimizes the population conditional average; for one unit the corrected objective is A. Oracle and independent pointwise-unbiased estimators supply the vector condition on its ordinary L1 domain.

", "headline": "Propositions 6 and 7, corrected: the exact expected reward of the zero-on-empty average rewards wATOw_{\\mathrm{ATO}}wATO​ and wATTw_{\\mathrm{ATT}}wATT​ and the finite-nnn best response (replaces the first display of each proof and both best-response rules)", "location": "PDF p. 15 (Proposition 6); PDF p. 16 (Proposition 7); PDF pp. 4-5 (Section 3)" }, "id": "P006-T2", "text": { "body": "PDF pp. 15-16 (Propositions 6 and 7); PDF pp. 4-5 (Section 3). The zero-on-empty ATO and conditionally unbiased ATT rewards have exact expectation h_n(q) A, empty probability (1-q)^n, and conditional means A/q when q>0. Their best responses maximize this finite-sample objective. On finite support the fixed-mass frontier is attained by a fractional threshold, and maximizing it over q gives an attained threshold optimum. Such an optimum can treat a nonmaximal score. The printed positive-maximizer rule optimizes the population conditional average; for one unit the corrected objective is A. Oracle and independent pointwise-unbiased estimators supply the vector condition on its ordinary L1 domain.", "headline": "Propositions 6 and 7, corrected: the exact expected reward of the zero-on-empty average rewards $w_{\\mathrm{ATO}}$ and $w_{\\mathrm{ATT}}$ and the finite-$n$ best response (replaces the first display of each proof and both best-response rules)", "location": "PDF p. 15 (Proposition 6); PDF p. 16 (Proposition 7); PDF pp. 4-5 (Section 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T2" }, { "declarations": [ { "claim_revision_id": "catalog:47b4ee1e004b9ca5f76359a6d38084481fd7ef28d301f3c167aefb43fd7e487f", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"GP006B2b.lean:463 takes Standing E, whose integrable_units field at P006_CounterfactualMetrics.lean:193-195 requires the evaluated estimator to be integrable for every admissible policy, including policies outside PiSet. Corrected T3 imposes the full-vector estimator conditional mean only for policies in the feasible set, and its inherited ordinary conditional-expectation domain therefore does not imply integrability under unavailable policies. The reviewer's geometric-covariate example works: independent auxiliary coupling under the feasible zero/one policies gives an integrable unbiased evaluation, while coupling the auxiliary state to X under an unavailable half-treatment policy makes the same evaluation nonintegrable. The attained-maximizer witnesses and inherited discrete covariate setting do not supply that missing global assumption.\", 'suggested_fix': 'Replace Standing by a version whose estimator-integrability requirement is restricted to policies in PiSet, retaining full evaluated-estimator integrability there for VectorUnbiased; impose outcome/design conditions only where needed. Keep the existing zero-policy and attained-maximizer hypotheses.', 'second_review': 'P006/Wave1/Gap/GP006B2b.lean:463 assumes Standing E, whose integrable_units field at P006/P006_CounterfactualMetrics.lean:193 requires evaluated-estimator integrability for every admissible policy; corrected T3 requires the full-vector estimator condition and its ordinary integrability domain only for policies in Π. This is an extra hypothesis, but its excluded cases are recovered by replacing E.μhat with zero outside Π, since no other Standing field depends on that estimator. This elementary extension preserves VectorUnbiased on Π and every feasible expectedATT, welfare value, and optimizer, so localizing the integrability field is a minor statement repair.', 'severity': 'minor'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.Wave1.T3_corrected_ATT_regret_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B2b.lean#L463", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:47b4ee1e004b9ca5f76359a6d38084481fd7ef28d301f3c167aefb43fd7e487f", "conditional_on": [], "credit_reason": "The witness supplies both nonemptiness and boundedness, so no real-supremum junk value is used.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.bestValue_eq", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L391", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:47b4ee1e004b9ca5f76359a6d38084481fd7ef28d301f3c167aefb43fd7e487f", "conditional_on": [], "credit_reason": "Includes q=0 and q=1; the positive sample-size condition matches the claim.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.emptyFactor_ge_one", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L705", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:47b4ee1e004b9ca5f76359a6d38084481fd7ef28d301f3c167aefb43fd7e487f", "conditional_on": [], "credit_reason": "The experiment theorem already exports the abstract positive-factor regret consequence.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.positive_factor_regret", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L715", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:47b4ee1e004b9ca5f76359a6d38084481fd7ef28d301f3c167aefb43fd7e487f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T3" }, "html": { "body": "

PDF p. 5 (Proposition 2); PDF pp. 17-18 (Appendix B.4). For every feasible class containing zero, vector-conditionally unbiased ATT has expected reward h_n(q)V and h_n(q)>=1. Zero has reward and welfare zero. Every reward best response has nonnegative reward and welfare, hence regret at most the finite supremum of feasible welfare. An attained welfare maximum gives the stated maximum formula. Per-policy identities require no maximizers and no estimator moments outside the feasible class. In the two-type example the actual regret is 1/40, whereas the printed closed form is 9/20.

", "headline": "Proposition 2 (ATT Regret), with the proof replaced (the printed proof substitutes the rule of Proposition 7)", "location": "PDF p. 5 (Section 3); PDF pp. 17-18 (Appendix B.4, restatement and proof)" }, "id": "P006-T3", "text": { "body": "PDF p. 5 (Proposition 2); PDF pp. 17-18 (Appendix B.4). For every feasible class containing zero, vector-conditionally unbiased ATT has expected reward h_n(q)V and h_n(q)>=1. Zero has reward and welfare zero. Every reward best response has nonnegative reward and welfare, hence regret at most the finite supremum of feasible welfare. An attained welfare maximum gives the stated maximum formula. Per-policy identities require no maximizers and no estimator moments outside the feasible class. In the two-type example the actual regret is 1/40, whereas the printed closed form is 9/20.", "headline": "Proposition 2 (ATT Regret), with the proof replaced (the printed proof substitutes the rule of Proposition 7)", "location": "PDF p. 5 (Section 3); PDF pp. 17-18 (Appendix B.4, restatement and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T3" }, { "declarations": [ { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "conditional_on": [], "credit_reason": "This is a witness result, not a proof of the affirmative alignment claims. The conditional-mean failure follows immediately from the bounded diagonal estimator value; row 04 transfers it to ATT and row 02 adds the constant-U bound failure.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T4_unbiasedness_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L446", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "conditional_on": [], "credit_reason": "The declaration supplies the numerical A5 component; reward optimality and unbiasedness are furnished by row 01, so it is not being treated as a stand-alone full refutation theorem.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T4_zero_heterogeneity_bound_failure", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2461", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "conditional_on": [], "credit_reason": "The specialized witness formula supplies the actual expected rewards; the general one-unit expansion is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.oneTT_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L400", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/P006_CounterfactualMetrics.lean:434 states all four reward and welfare formulas in T4(A3) and T4(B2), with the random estimators retained through rewardA and rewardB. The formulas agree with the target on the entire feasible unit square. Their validity as polynomial identities outside that square is a harmless extension of the finite-sum definitions.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.witnesses_AB_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L434", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "conditional_on": [], "credit_reason": "Auxiliary realized-reward equality, including the explicit zero-on-empty convention.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.wATT_one_eq_TT", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2105", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:d7110f1d806b458a3d3de21c188e67cb78050597ffc7e38119cffba6707ff9ef", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T4" }, "html": { "body": "

PDF p. 4 (Assumption 1); PDF p. 16 (proof of Proposition 7); PDF p. 17 (proof of Proposition 9); PDF p. 18 (Proposition 3); PDF p. 19 (Theorem 1); PDF p. 22 (Theorem 3 and proof); PDF pp. 5 and 17 (Proposition 2) Both canonical pointwise-unbiased estimator witnesses under Assumption 1 alone have the stated evaluation biases, actual ATT/TT reward maxima, and positive welfare regrets, including zero private heterogeneity.

", "headline": "Assumption 1 and the inference drawn from it in the proofs of Propositions 7 and 9 and of Theorem 3; Propositions 2, 3, 7 and 9, Theorem 1 and Theorem 3 under Assumption 1 alone", "location": "PDF p. 4 (Assumption 1); PDF p. 16 (proof of Proposition 7); PDF p. 17 (proof of Proposition 9); PDF p. 18 (Proposition 3); PDF p. 19 (Theorem 1); PDF p. 22 (Theorem 3 and proof); PDF pp. 5 and 17 (Proposition 2)" }, "id": "P006-T4", "text": { "body": "PDF p. 4 (Assumption 1); PDF p. 16 (proof of Proposition 7); PDF p. 17 (proof of Proposition 9); PDF p. 18 (Proposition 3); PDF p. 19 (Theorem 1); PDF p. 22 (Theorem 3 and proof); PDF pp. 5 and 17 (Proposition 2) Both canonical pointwise-unbiased estimator witnesses under Assumption 1 alone have the stated evaluation biases, actual ATT/TT reward maxima, and positive welfare regrets, including zero private heterogeneity.", "headline": "Assumption 1 and the inference drawn from it in the proofs of Propositions 7 and 9 and of Theorem 3; Propositions 2, 3, 7 and 9, Theorem 1 and Theorem 3 under Assumption 1 alone", "location": "PDF p. 4 (Assumption 1); PDF p. 16 (proof of Proposition 7); PDF p. 17 (proof of Proposition 9); PDF p. 18 (Proposition 3); PDF p. 19 (Theorem 1); PDF p. 22 (Theorem 3 and proof); PDF pp. 5 and 17 (Proposition 2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T4" }, { "declarations": [ { "claim_revision_id": "catalog:4860e19b9369c0e87d7de7a8557bc09a43cbe4daa60818d5b7603d7adae5ac1a", "conditional_on": [], "credit_reason": "This is the exact witness under printed pointwise unbiasedness, not a refutation under the independently sampled-estimator repair.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T5_ranking_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L531", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4860e19b9369c0e87d7de7a8557bc09a43cbe4daa60818d5b7603d7adae5ac1a", "conditional_on": [], "credit_reason": "Harmless algebraic extension; on the feasible square these are genuine unconditional scores.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.ranking_witness_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L521", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:4860e19b9369c0e87d7de7a8557bc09a43cbe4daa60818d5b7603d7adae5ac1a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T5" }, "html": { "body": "

PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof: 'As shown in the proof of Theorem 1') The canonical shared-estimator witness C reverses both ranking orders claimed for the two specified agents under pointwise Assumption 1 alone.

", "headline": "Theorem 2 (Ranking Desiderata Satisfied) under Assumption 1 alone: a shared pointwise-unbiased estimator reverses both ranking properties", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof: 'As shown in the proof of Theorem 1')" }, "id": "P006-T5", "text": { "body": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof: 'As shown in the proof of Theorem 1') The canonical shared-estimator witness C reverses both ranking orders claimed for the two specified agents under pointwise Assumption 1 alone.", "headline": "Theorem 2 (Ranking Desiderata Satisfied) under Assumption 1 alone: a shared pointwise-unbiased estimator reverses both ranking properties", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof: 'As shown in the proof of Theorem 1')" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T5" }, { "declarations": [ { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/Wave1/Gap/RepairT6T7.lean:380 establishes T6(i)-(iii): expected TT equals n times welfare, the maximizer sets coincide on every feasible class, best responses have zero regret, and the unconstrained positive-effect rule is optimal. Its policy-specific Design weakens the standing model rather than restricting it. Integrability is required only for the treated residual whose ordinary conditional expectation is stipulated in Assumption 1-prime, so it adds no substantive condition to that corrected premise.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T6_corrected_TT_minimal_integrability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/RepairT6T7.lean#L348", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"RepairT6T7.lean:493 requires hR at lines 497-498: the treated estimator residual must be globally integrable for every feasible policy and unit. T6(iv)'s stated sufficient premises are independence of the random estimator and pointwise unbiasedness, which do not imply hR: with P(X=k)=2^(-k-1), an independent symmetric sign Z, zero potential outcomes, pi=1, and hat_mu(x)=2^x Z, every fixed-query estimate is integrable and unbiased but the treated residual has infinite absolute mean. This is a substantive missing case in the stated implication, rather than integrability on irrelevant untreated observations. The Lean guard is needed to give the conclusion its ordinary conditional-expectation meaning; dropping it would permit Mathlib's zero convention for nonintegrable inputs, not repair the mathematical implementation claim.\", 'suggested_fix': 'Correct T6(iv) to require E[abs(T_i^pi*(hat_mu_0(X_i)-mu_0(X_i))) ] < infinity for every feasible policy and unit, or an explicit stronger envelope condition implying it. Keep hR in the Lean theorem and describe independence plus pointwise unbiasedness as sufficient on that integrability domain.', 'second_review': \"P006/Wave1/Gap/RepairT6T7.lean:497 explicitly assumes integrability of the treated estimator residual, while T6(iv) claims independence and fixed-query unbiasedness suffice. Take P(X=k)=2^(-k-1) for k≥0, independent symmetric Z in {-1,1}, zero potential outcomes, π=1, and μhat(x)=2^x Z: every fixed-query estimate is integrable and unbiased, but E|T μhat(X)| is infinite. Thus the target's sufficient-implementation correction needs a treated-residual integrability premise, which the Lean theorem appropriately retains. Removing that guard would invoke Mathlib's zero convention for nonintegrable conditional-expectation inputs (MeasureTheory/Function/ConditionalExpectation/Basic.lean:102), rather than supply the ordinary expectation claimed.\", 'severity': None}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T6_independent_estimation_minimal_integrability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/RepairT6T7.lean#L456", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "Finite specialization or auxiliary reduction not needed: the two credited general measurable-experiment results establish the stated target.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T6_finite_reward_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1963", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "Finite specialization or auxiliary reduction not needed: the two credited general measurable-experiment results establish the stated target.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T6_finite_corrected_alignment", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2008", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "Finite specialization or auxiliary reduction not needed: the two credited general measurable-experiment results establish the stated target.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T6_finite_aux_actual_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2372", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "CT-01:P006-T6:accepted", "conditional_on": [], "credit_reason": "Finite specialization or auxiliary reduction not needed: the two credited general measurable-experiment results establish the stated target.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T6_finite_auxiliary_estimation", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2438", "status": "proved", "support_scope": "limited", "used_in_solution": false } ], "formalization": { "claim_revision_id": "CT-01:P006-T6:accepted", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T6" }, "html": { "body": "

PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof) Corrected treated-evaluation unbiasedness aligns TT with welfare on every feasible class, with the positive-effect rule in the unconstrained class and the stated independent-estimation implementation.

", "headline": "Assumption 1, corrected to unbiasedness at the treated evaluation observations; Proposition 9 (TT Best Response) and Proposition 3 (TT Regret) under it", "location": "PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof)" }, "id": "P006-T6", "text": { "body": "PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof) Corrected treated-evaluation unbiasedness aligns TT with welfare on every feasible class, with the positive-effect rule in the unconstrained class and the stated independent-estimation implementation.", "headline": "Assumption 1, corrected to unbiasedness at the treated evaluation observations; Proposition 9 (TT Best Response) and Proposition 3 (TT Regret) under it", "location": "PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T6" }, { "declarations": [ { "claim_revision_id": "catalog:90860792c7776082f25c4f403f2c2b32522ad914806678a13e3fb4b17c442a2e", "conditional_on": [], "credit_reason": "Every-maximizer characterization, actual welfare and zero regret are all exported. The stronger integrability assumptions mentioned in the historical record are expressly in the corrected T7 target itself.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T7_corrected_weighted_alignment_treated_integrability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/DecideF.lean#L95", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:90860792c7776082f25c4f403f2c2b32522ad914806678a13e3fb4b17c442a2e", "conditional_on": [], "credit_reason": "The general measurable-experiment theorem covers the canonical target; this finite specialization is not needed.", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T7_finite_positive_rule", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2062", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "catalog:90860792c7776082f25c4f403f2c2b32522ad914806678a13e3fb4b17c442a2e", "conditional_on": [], "credit_reason": "The full weighted-alignment theorem is used directly; its supporting pointwise inequality is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.pointwise_policy_gain_le", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2050", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:90860792c7776082f25c4f403f2c2b32522ad914806678a13e3fb4b17c442a2e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T7" }, "html": { "body": "

PDF p. 6 (Section 4); PDF p. 19 (Appendix C.2, restatement and proof) Strictly positive weighted TT aligns every maximizing policy with the welfare optimum under the corrected moment and integrability conditions.

", "headline": "Theorem 1 (Incentive Alignment), under the corrected Assumption 1 and with a strictly positive weight", "location": "PDF p. 6 (Section 4); PDF p. 19 (Appendix C.2, restatement and proof)" }, "id": "P006-T7", "text": { "body": "PDF p. 6 (Section 4); PDF p. 19 (Appendix C.2, restatement and proof) Strictly positive weighted TT aligns every maximizing policy with the welfare optimum under the corrected moment and integrability conditions.", "headline": "Theorem 1 (Incentive Alignment), under the corrected Assumption 1 and with a strictly positive weight", "location": "PDF p. 6 (Section 4); PDF p. 19 (Appendix C.2, restatement and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T7" }, { "declarations": [ { "claim_revision_id": "source-claim-import-2026-11-08:P006-T8:1", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"GP006B3.lean:497 assumes Standing E, whose integrable_units field at P006_CounterfactualMetrics.lean:193-195 requires the full estimator evaluation to be integrable under every admissible policy. Corrected T8 requires its treated residual moment only on PiSet, so it permits a nonintegrable estimator on untreated observations and imposes no estimator moment under unavailable policies. For example, take n=1, P(X=k)=2^(-k-1), singleton U, independent symmetric Z, hat_mu(x)=2^x Z, Y0=0, Y1=1, and feasible policies zero and 1{x=0}; all treated residuals are integrable and centered, both optima are attained, and gamma=0, but the full estimator evaluation is not integrable. Maximizer witnesses are appropriate for the regret conclusion, and singleton-class specialization recovers the policy identities on the theorem's stronger domain, so optimizer packaging is not a separate material defect.\", 'suggested_fix': \"Replace Standing's estimator requirement by integrability of the treated residual on feasible policies, leaving the marginal baseline and potential outcomes integrable. Export the per-policy identities under those local assumptions and then derive the regret bound for supplied maximizers.\", 'second_review': \"P006/Wave1/Gap/GP006B3.lean:499 assumes Standing E, whose integrable_units field at P006/P006_CounterfactualMetrics.lean:193 requires full estimator evaluations to be integrable under every admissible policy. Corrected T8 instead imposes the treated residual moment conditional on (X,U) only for feasible policies. With n=1, P(X=k)=2^(-k-1), singleton U, independent symmetric Z, μhat(x)=2^x Z, Y0=0, Y1=1 and Π={0,1{x=0}}, the target's treated residuals are integrable and centered, both maxima are attained and γ=0, yet the full estimator evaluation is nonintegrable. This is a substantive excluded case, requiring estimator integrability to be localized to treated residuals on feasible policies.\", 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.Wave1.T8_corrected_information_bound_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B3.lean#L382", "status": "proved", "support_scope": "limited", "used_in_solution": false }, { "claim_revision_id": "source-claim-import-2026-11-08:P006-T8:1", "conditional_on": [], "credit_reason": "Final status wrong_credit: not usable for this target; no statement or proof is taken from this row.", "credit_status": "wrong_credit", "fidelity_status_baseline": "wrong_credit", "lean_name": "AronowAudit.P006.Wave1.T8_lemma2_proposition10_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B3.lean#L515", "status": "proved", "support_scope": "withdrawn", "used_in_solution": false }, { "claim_revision_id": "source-claim-import-2026-11-08:P006-T8:1", "conditional_on": [], "credit_reason": "The experiment theorem already exports the regret bound, so the abstract decision helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.uniform_error_regret", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L777", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "source-claim-import-2026-11-08:P006-T8:1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T8" }, "html": { "body": "

PDF p. 7 (Section 5); PDF pp. 21-22 (Appendix D.2: Lemma 2, Proposition 10, Theorem 3 and proof). Under the usual measure-theoretic conditions, treated conditional unbiasedness at the marginal untreated baseline gives the normalized TT identity, the exact marginal-heterogeneity bias identity and bound, and regret at most twice the marginal heterogeneity for every reward and welfare maximizing pair in any feasible class. The selected conditional-mean formulation is equivalent, and independent pointwise-unbiased auxiliary data suffice on the treated-residual integrability domain. Qualified Lemma 2 is a sum over units; its n-times form for every fixed unit is equivalent to equality of weighted evaluation means, which a common joint evaluation law supplies. Conditional independence implies the required weighted moment, equivalent to factorization when the full evaluation is integrable. Qualified Proposition 10 gives the uniform absolute error and regret/triangle bounds for a common evaluation law, including the derived independent auxiliary-data case, without unbiasedness. All estimator moments are confined to feasible policies; the main clauses require no integrability on untreated observations. Maximizers are needed only for regret.

", "headline": "Theorem 3 (Regret With Information Asymmetry), under the corrected Assumption 1; with Lemma 2 and Proposition 10, on which its proof rests", "location": "PDF p. 7 (Section 5); PDF pp. 21-22 (Appendix D.2: Lemma 2, Proposition 10, Theorem 3 and proof)" }, "id": "P006-T8", "text": { "body": "PDF p. 7 (Section 5); PDF pp. 21-22 (Appendix D.2: Lemma 2, Proposition 10, Theorem 3 and proof). Under the usual measure-theoretic conditions, treated conditional unbiasedness at the marginal untreated baseline gives the normalized TT identity, the exact marginal-heterogeneity bias identity and bound, and regret at most twice the marginal heterogeneity for every reward and welfare maximizing pair in any feasible class. The selected conditional-mean formulation is equivalent, and independent pointwise-unbiased auxiliary data suffice on the treated-residual integrability domain. Qualified Lemma 2 is a sum over units; its n-times form for every fixed unit is equivalent to equality of weighted evaluation means, which a common joint evaluation law supplies. Conditional independence implies the required weighted moment, equivalent to factorization when the full evaluation is integrable. Qualified Proposition 10 gives the uniform absolute error and regret/triangle bounds for a common evaluation law, including the derived independent auxiliary-data case, without unbiasedness. All estimator moments are confined to feasible policies; the main clauses require no integrability on untreated observations. Maximizers are needed only for regret.", "headline": "Theorem 3 (Regret With Information Asymmetry), under the corrected Assumption 1; with Lemma 2 and Proposition 10, on which its proof rests", "location": "PDF p. 7 (Section 5); PDF pp. 21-22 (Appendix D.2: Lemma 2, Proposition 10, Theorem 3 and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T8" }, { "declarations": [ { "claim_revision_id": "catalog:9f9232d0db1867a9a71b8bb1b0cb295af08da18251e1dfcd353784ab6895e216", "conditional_on": [], "credit_reason": "The restrictions n>0 and k>0 are part of the corrected finite fitted-effect claim, and the strict inequality is conditional on kPDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset'); Table 1, PDF p. 9, row wTTw_{\\mathrm{TT}}wTT​ The fitted whole-sample and beneficiary means obey the exact denominator identity, numerical intervals, printed-table bounds, and three-decimal rounding intervals.

", "headline": "Section 6, the two sentences 'On the horses that would benefit, the average benefit was 0.147' and 'On the patients that would benefit, the average benefit of treatment was 0.025', corrected: the two numbers are averages over the whole analysis sample", "location": "PDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset'); Table 1, PDF p. 9, row wTTw_{\\mathrm{TT}}wTT​" }, "id": "P006-T9", "text": { "body": "PDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset'); Table 1, PDF p. 9, row $w_{\\mathrm{TT}}$ The fitted whole-sample and beneficiary means obey the exact denominator identity, numerical intervals, printed-table bounds, and three-decimal rounding intervals.", "headline": "Section 6, the two sentences 'On the horses that would benefit, the average benefit was 0.147' and 'On the patients that would benefit, the average benefit of treatment was 0.025', corrected: the two numbers are averages over the whole analysis sample", "location": "PDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset'); Table 1, PDF p. 9, row $w_{\\mathrm{TT}}$" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T9" }, { "declarations": [ { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "conditional_on": [], "credit_reason": "Auxiliary rows R044-02 through R044-05 complete the ATO and full-law invariance checks. Unit is a harmless relabeling of the constant covariate 0.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T10_singleton_feasible_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1204", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "conditional_on": [], "credit_reason": "The exact one-unit expectation is used directly; the count helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.treatedCount_one", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1136", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "conditional_on": [], "credit_reason": "The actual ATO reward occurs in the expectation definition; its exported expectation formula supplies the needed reduction.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.wATO_one", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1142", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "conditional_on": [], "credit_reason": "Outside admissible probabilistic instances, coinMass is an algebraic weight and nonintegrable real integrals are totalized to zero; no such convention is needed for the target specialization.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.oneATOExpectation_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1171", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "conditional_on": [], "credit_reason": "For finite-outcome probability laws and [0,1]-valued π the reward is integrable, so the target use is an ordinary expectation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.oneATO_invariance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1179", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:ff6b0ab82da50fcde0776e15c25106b9c255be6ca4006b27eeda51174a564778", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T10", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T10" }, "html": { "body": "

PDF p. 14 (Appendix A.2: Assumption 3, Theorem 4 and proof) The canonical singleton always-treated counterexample has identically zero regret despite nondegeneracy and treated-data invariance, contradicting the paper's printed impossibility statement and its withholding step. The general one-unit expectation identities assume finite-moment probability laws and admissible policies.

", "headline": "Theorem 4 (Appendix A.2, necessity of estimating the untreated potential outcome), as printed, and the step 'Choose β>α\\beta>\\alphaβ>α. Then the set of welfare maximizing policies contains a π∗\\pi^*π∗ with π∗(xˉ)=0\\pi^*(\\bar x)=0π∗(xˉ)=0' of its proof", "location": "PDF p. 14 (Appendix A.2: Assumption 3, Theorem 4 and proof)" }, "id": "P006-T10", "text": { "body": "PDF p. 14 (Appendix A.2: Assumption 3, Theorem 4 and proof) The canonical singleton always-treated counterexample has identically zero regret despite nondegeneracy and treated-data invariance, contradicting the paper's printed impossibility statement and its withholding step. The general one-unit expectation identities assume finite-moment probability laws and admissible policies.", "headline": "Theorem 4 (Appendix A.2, necessity of estimating the untreated potential outcome), as printed, and the step 'Choose $\\beta>\\alpha$. Then the set of welfare maximizing policies contains a $\\pi^*$ with $\\pi^*(\\bar x)=0$' of its proof", "location": "PDF p. 14 (Appendix A.2: Assumption 3, Theorem 4 and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T10" }, { "declarations": [ { "claim_revision_id": "catalog:6973b4ca7980898bb52175502fd78566b466bb4e9ae82205c87b89c48e8c6b07", "conditional_on": [], "credit_reason": "The abstract functional neither assumes nor defines the regret conclusion. The feasible zero policy supplies an attained maximum welfare of zero in the shifted family.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T11_corrected_impossibility", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1255", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6973b4ca7980898bb52175502fd78566b466bb4e9ae82205c87b89c48e8c6b07", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T11_unbounded_shift_arithmetic", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1351", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6973b4ca7980898bb52175502fd78566b466bb4e9ae82205c87b89c48e8c6b07", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T11", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T11" }, "html": { "body": "

PDF p. 14 (Appendix A.2) Feasible zero treatment and treated-data invariance permit unbounded regret for some best response through the explicit shifted-law construction, while the zero-reward tie example rules out a conclusion for every best response.

", "headline": "Theorem 4, corrected: the policy that treats nobody is feasible, and the conclusion is stated for some best response", "location": "PDF p. 14 (Appendix A.2)" }, "id": "P006-T11", "text": { "body": "PDF p. 14 (Appendix A.2) Feasible zero treatment and treated-data invariance permit unbounded regret for some best response through the explicit shifted-law construction, while the zero-reward tie example rules out a conclusion for every best response.", "headline": "Theorem 4, corrected: the policy that treats nobody is feasible, and the conclusion is stated for some best response", "location": "PDF p. 14 (Appendix A.2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T11" }, { "declarations": [ { "claim_revision_id": "catalog:dd1b525596eb0a23d5c39b57d9b87fa54efacc880464ad2baed76f02f3ffcf3a", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P006/P006_CounterfactualMetrics.lean:1617 states the target's two probability laws, their identical support, the printed absolute-continuity direction, and the atom that prevents a density in the required reverse direction. Representing the laws on ambient Borel R while proving support [0,1] preserves the counterexample on the target's covariate space. The row covers the density-existence obstruction and does not claim to export the separate two-agent completion in T12(d).\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T12_density_direction_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1383", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dd1b525596eb0a23d5c39b57d9b87fa54efacc880464ad2baed76f02f3ffcf3a", "conditional_on": [], "credit_reason": "The density counterexample already exports both support equalities, so the generic scaling helper is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.smul_support_eq", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1367", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:dd1b525596eb0a23d5c39b57d9b87fa54efacc880464ad2baed76f02f3ffcf3a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T12" }, "html": { "body": "

PDF p. 6 (Theorem 2); PDF p. 20 (Appendix C.2). Counterexample to the sufficiency reading of the printed absolute-continuity direction: P_agent << P_reference alone need not supply the named reference-to-agent density. Lebesgue probability on [0,1] and its half-Lebesgue, half-atom reference have the same support and satisfy the printed direction, but the reference has mass 1/2 at the Lebesgue-null singleton {1/2} and has no representing density. Complete both agents with one unit, independent uniform treatment seeds, unconstrained policies, potential outcomes 0 and 1, common zero baseline and oracle estimator. Every other stochastic-model condition holds and both treatment effects are identically one. This does not refute the ranking conclusion under a separate assumption that the named representing density exists.

", "headline": "Theorem 2 (Ranking Desiderata Satisfied), first sentence: under the printed condition 'PXkP_{X^k}PXk​ absolutely continuous with respect to PX0P_{X_0}PX0​​' the derivative dPX0/dPXkdP_{X_0}/dP_{X^k}dPX0​​/dPXk​ named in the theorem need not exist", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, Appendix C.2)" }, "id": "P006-T12", "text": { "body": "PDF p. 6 (Theorem 2); PDF p. 20 (Appendix C.2). Counterexample to the sufficiency reading of the printed absolute-continuity direction: P_agent << P_reference alone need not supply the named reference-to-agent density. Lebesgue probability on [0,1] and its half-Lebesgue, half-atom reference have the same support and satisfy the printed direction, but the reference has mass 1/2 at the Lebesgue-null singleton {1/2} and has no representing density. Complete both agents with one unit, independent uniform treatment seeds, unconstrained policies, potential outcomes 0 and 1, common zero baseline and oracle estimator. Every other stochastic-model condition holds and both treatment effects are identically one. This does not refute the ranking conclusion under a separate assumption that the named representing density exists.", "headline": "Theorem 2 (Ranking Desiderata Satisfied), first sentence: under the printed condition '$P_{X^k}$ absolutely continuous with respect to $P_{X_0}$' the derivative $dP_{X_0}/dP_{X^k}$ named in the theorem need not exist", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, Appendix C.2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T12" }, { "declarations": [ { "claim_revision_id": "catalog:7ae8a9696bb03ad3c469a84ccca02d1346ff5126779e40a148ccef196c914380", "conditional_on": [], "credit_reason": "The qualification is in claims.json itself, not merely disclosed in a row description; thus hR and hreward are not extra assumptions against the current corrected target. The printed unqualified theorem is a different assertion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T13_corrected_ranking_treated_residuals", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/Fid2P006.lean#L272", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:7ae8a9696bb03ad3c469a84ccca02d1346ff5126779e40a148ccef196c914380", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T13" }, "html": { "body": "

PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof) The corrected normalized reference density gives the score identity, change of measure, attained optimal scores, integrated positive-part comparison, and both specified ranking properties.

", "headline": "Theorem 2 (Ranking Desiderata Satisfied), corrected: 'PX0P_{X_0}PX0​​ absolutely continuous with respect to PX(k)P_{X^{(k)}}PX(k)​' (printed in the reverse direction)", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof)" }, "id": "P006-T13", "text": { "body": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof) The corrected normalized reference density gives the score identity, change of measure, attained optimal scores, integrated positive-part comparison, and both specified ranking properties.", "headline": "Theorem 2 (Ranking Desiderata Satisfied), corrected: '$P_{X_0}$ absolutely continuous with respect to $P_{X^{(k)}}$' (printed in the reverse direction)", "location": "PDF p. 6 (Theorem 2); PDF p. 20 (restatement, equation (2), proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T13" }, { "declarations": [ { "claim_revision_id": "catalog:5885ed05d861e2f9284e9234331a816864f53a722eb18f9f2ce71495bdad950e", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"P006/P006_CounterfactualMetrics.lean:1013 requires both reference-law positive parts to be integrable (hj and hk at lines 1015-1016), whereas corrected T14(f) asserts the expectation ordering from pointwise dominance without those restrictions. S9 gives finite outcome means under the agents' own laws, not under P0. For example, on m≥0 take both agent laws with masses 3/4^(m+1), reference masses 1/2^(m+1), τj(m)=2^(m+1) and τk(m)=1: both effects have finite agent-law means, but the reference positive-part means are infinity and 1. This nontrivial case satisfies the intended extended expectation comparison but is excluded by the Lean hypotheses; permitting arbitrary measures instead of probability measures does not repair that exclusion.\", 'suggested_fix': 'State the comparison using nonnegative extended integrals of ENNReal.ofReal (positivePart (τ x)) and drop hj and hk; retain the real-integral version as its finite-expectation corollary.', 'second_review': \"P006/P006_CounterfactualMetrics.lean:1013 requires both positive parts to be integrable under the reference measure, through hj and hk at lines 1015–1016. Corrected T14(f) requires the reference-expectation ordering from pointwise dominance, whereas S9 supplies finite outcome means under the agents' own population laws only. Changing to a common-support reference law can produce an infinite positive-part expectation, so these are substantive excluded cases and the comparison should also cover extended nonnegative expectations.\", 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T14_corrected_rank_inequality", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L830", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:5885ed05d861e2f9284e9234331a816864f53a722eb18f9f2ce71495bdad950e", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.positivePart_eq_max", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L821", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:5885ed05d861e2f9284e9234331a816864f53a722eb18f9f2ce71495bdad950e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T14", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T14" }, "html": { "body": "

PDF p. 5 (Definitions 1 and 2); PDF pp. 19-20 (score corrections). Pointwise treatment-effect dominance orders reference positive-part expectations in [0,infinity], without a finiteness assumption. When both positive parts are integrable, the same comparison holds for real expectations. Each corrected ranking compares its own agent's score, and the weighted reward evaluates the unit's own weight inside the sum; the separate score identities belong to T30, T7 and T13 as specified by the canonical claim.

", "headline": "Definitions 1 and 2 (rank preservation), equation (2), and the score displays in the proofs of Proposition 4, Theorem 1 and Theorem 2, with each agent's own arguments and indices (printed with Xk\\mathbf X^kXk in agent jjj's score, Y(k)\\mathbf Y^{(k)}Y(k) in agent jjj's score, gk(X(k))g_k(X^{(k)})gk​(X(k)), the arguments in the order (Y,Tπ,X)(\\mathbf Y,\\mathbf T^\\pi,\\mathbf X)(Y,Tπ,X), and kkk on both sides of an inequality)", "location": "PDF p. 5 (Definitions 1 and 2); PDF p. 19 (proofs of Proposition 4 and Theorem 1); PDF p. 20 (equation (2) and proof of Theorem 2)" }, "id": "P006-T14", "text": { "body": "PDF p. 5 (Definitions 1 and 2); PDF pp. 19-20 (score corrections). Pointwise treatment-effect dominance orders reference positive-part expectations in [0,infinity], without a finiteness assumption. When both positive parts are integrable, the same comparison holds for real expectations. Each corrected ranking compares its own agent's score, and the weighted reward evaluates the unit's own weight inside the sum; the separate score identities belong to T30, T7 and T13 as specified by the canonical claim.", "headline": "Definitions 1 and 2 (rank preservation), equation (2), and the score displays in the proofs of Proposition 4, Theorem 1 and Theorem 2, with each agent's own arguments and indices (printed with $\\mathbf X^k$ in agent $j$'s score, $\\mathbf Y^{(k)}$ in agent $j$'s score, $g_k(X^{(k)})$, the arguments in the order $(\\mathbf Y,\\mathbf T^\\pi,\\mathbf X)$, and $k$ on both sides of an inequality)", "location": "PDF p. 5 (Definitions 1 and 2); PDF p. 19 (proofs of Proposition 4 and Theorem 1); PDF p. 20 (equation (2) and proof of Theorem 2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T14" }, { "declarations": [ { "claim_revision_id": "catalog:e093b74fe05eea9fddd9a39e9676449c88f31347d85e7773a351192cc615fa1d", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T15_untreated_positivity_counterexamples", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1471", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:e093b74fe05eea9fddd9a39e9676449c88f31347d85e7773a351192cc615fa1d", "conditional_on": [], "credit_reason": "Singleton private U is deliberately the counterexample construction. The zero conclusion follows from actual zero within-x differences, not from a nonintegrable-function convention.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T15_constant_private_heterogeneity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2475", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:e093b74fe05eea9fddd9a39e9676449c88f31347d85e7773a351192cc615fa1d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T15" }, "html": { "body": "

PDF p. 23 (Appendix D.3, first paragraph; definition of μ^0π\\hat\\mu_0^\\piμ^​0π​; Theorem 5); PDF p. 24 (proof of Theorem 5; Lemma 3) Both canonical positive-treatment counterexamples establish failure of untreated identification and of positive treatment alone to supply the untreated mass needed for the conditional-mean construction and Lemma 3; their singleton-private-covariate maximum heterogeneity is zero.

", "headline": "Appendix D.3: the positivity condition 'π(x)>0\\pi(x)>0π(x)>0' of the first paragraph and 'π(x,u)>0\\pi(x,u)>0π(x,u)>0' of Theorem 5 and its proof (the estimate μ^0π\\hat\\mu_0^\\piμ^​0π​ is built from untreated units)", "location": "PDF p. 23 (Appendix D.3, first paragraph; definition of μ^0π\\hat\\mu_0^\\piμ^​0π​; Theorem 5); PDF p. 24 (proof of Theorem 5; Lemma 3)" }, "id": "P006-T15", "text": { "body": "PDF p. 23 (Appendix D.3, first paragraph; definition of $\\hat\\mu_0^\\pi$; Theorem 5); PDF p. 24 (proof of Theorem 5; Lemma 3) Both canonical positive-treatment counterexamples establish failure of untreated identification and of positive treatment alone to supply the untreated mass needed for the conditional-mean construction and Lemma 3; their singleton-private-covariate maximum heterogeneity is zero.", "headline": "Appendix D.3: the positivity condition '$\\pi(x)>0$' of the first paragraph and '$\\pi(x,u)>0$' of Theorem 5 and its proof (the estimate $\\hat\\mu_0^\\pi$ is built from untreated units)", "location": "PDF p. 23 (Appendix D.3, first paragraph; definition of $\\hat\\mu_0^\\pi$; Theorem 5); PDF p. 24 (proof of Theorem 5; Lemma 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T15" }, { "declarations": [ { "claim_revision_id": "catalog:4494117a064a613ee4f2ae59937011930e3dfdaa5589108004bc84b7c9ae0148", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/Wave1/Gap/GLP006B1.lean:97 represents the conditional distribution P(U | X=x) appearing in corrected T16(i) by a Markov kernel and uses the stipulated positivity q_pi(x)>0 almost everywhere. The selected-baseline ratio, conditional-law normalization, extrema bounds, score-error bounds, and regret bound match the corrected population-mean claim. Fiber integrability follows from the standing finite mean and joint-law factorization, while the attained extrema and optimizer witnesses express the minima and maxima used in the target. These are representations of the corrected claim rather than a finite-sample estimator restriction.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T16_corrected_agent_data_bound_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GLP006B1.lean#L97", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4494117a064a613ee4f2ae59937011930e3dfdaa5589108004bc84b7c9ae0148", "conditional_on": [], "credit_reason": "This row covers the explicitly identified sufficient condition and full-information component; it does not replace the main target’s weaker a.e. condition by π<1 everywhere.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T16_sufficient_positivity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1523", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:4494117a064a613ee4f2ae59937011930e3dfdaa5589108004bc84b7c9ae0148", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.finite_convex_mean_bounds", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1621", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:4494117a064a613ee4f2ae59937011930e3dfdaa5589108004bc84b7c9ae0148", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T16", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T16" }, "html": { "body": "

PDF p. 23 (Appendix D.3, first paragraph; Assumption 4; Theorem 5); PDF p. 24 (proof; Lemma 3). Corrected untreated-arm positivity gives the conditional-law, selected-mean, bias and regret bounds, with the sufficient pointwise condition and full-information identity.

", "headline": "Theorem 5 and the first paragraph of Appendix D.3, corrected to positivity of the untreated arm; with Lemma 3", "location": "PDF p. 23 (Appendix D.3, first paragraph; Assumption 4; Theorem 5); PDF p. 24 (proof; Lemma 3)" }, "id": "P006-T16", "text": { "body": "PDF p. 23 (Appendix D.3, first paragraph; Assumption 4; Theorem 5); PDF p. 24 (proof; Lemma 3). Corrected untreated-arm positivity gives the conditional-law, selected-mean, bias and regret bounds, with the sufficient pointwise condition and full-information identity.", "headline": "Theorem 5 and the first paragraph of Appendix D.3, corrected to positivity of the untreated arm; with Lemma 3", "location": "PDF p. 23 (Appendix D.3, first paragraph; Assumption 4; Theorem 5); PDF p. 24 (proof; Lemma 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T16" }, { "declarations": [ { "claim_revision_id": "catalog:dff0f1b4dea4eef3efd11037a11e49f57f715d2feda3e6eaa03fdf71c1b99fb1", "conditional_on": [], "credit_reason": "The row covers clauses (b,c). finiteSelectedBaseline defines the ratio; this does not independently prove clause (a)’s equality with the actual observed-outcome conditional expectation for a general policy.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T17_selected_type", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1577", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dff0f1b4dea4eef3efd11037a11e49f57f715d2feda3e6eaa03fdf71c1b99fb1", "conditional_on": [], "credit_reason": "Fidelity sweep: ok", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T17_three_type_example", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1609", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dff0f1b4dea4eef3efd11037a11e49f57f715d2feda3e6eaa03fdf71c1b99fb1", "conditional_on": [], "credit_reason": "The row refutes an unrestricted measurable-space assertion, whereas canonical clause (d) positively asserts the finite conditional-type comparison; wrong_credit cannot supply that clause or change the target.", "credit_status": "wrong_credit", "fidelity_status_baseline": "wrong_credit", "lean_name": "AronowAudit.P006.Wave1.T17_maximum_dominates_marginal_printed_false", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/DecideE.lean#L836", "status": "counterexample_proved", "support_scope": "withdrawn", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:dff0f1b4dea4eef3efd11037a11e49f57f715d2feda3e6eaa03fdf71c1b99fb1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T17", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T17" }, "html": { "body": "

PDF p. 23 (Appendix D.3, paragraph after Assumption 4). For finite positive-mass conditional types, the actual untreated outcome mean is the probability-weighted selection ratio. Leaving only one type untreated selects its mean; treating only it selects the complementary mean (the three-type values are 0 and 2). Under the usual measure-theoretic conditions, for every conditional probability kernel, including infinite and nonatomic private spaces, the marginal deviation is integrable and bounded almost surely and in expectation by the maximum deviation. Thus Assumption 4 implies Assumption 2 with the same constant. Marginal heterogeneity alone cannot bound endogenous-baseline regret: it can equal 1 while regret exceeds every prescribed positive threshold.

", "headline": "Appendix D.3, the sentence 'Intuitively, Assumption 4 is not sufficient to bound the regret ... because the agent can choose π\\piπ to allocate treatment to single values of u~∈U\\tilde u\\in\\mathcal Uu~∈U', corrected: Assumption 2, and the selected type is the one left untreated", "location": "PDF p. 23 (Appendix D.3, paragraph after Assumption 4)" }, "id": "P006-T17", "text": { "body": "PDF p. 23 (Appendix D.3, paragraph after Assumption 4). For finite positive-mass conditional types, the actual untreated outcome mean is the probability-weighted selection ratio. Leaving only one type untreated selects its mean; treating only it selects the complementary mean (the three-type values are 0 and 2). Under the usual measure-theoretic conditions, for every conditional probability kernel, including infinite and nonatomic private spaces, the marginal deviation is integrable and bounded almost surely and in expectation by the maximum deviation. Thus Assumption 4 implies Assumption 2 with the same constant. Marginal heterogeneity alone cannot bound endogenous-baseline regret: it can equal 1 while regret exceeds every prescribed positive threshold.", "headline": "Appendix D.3, the sentence 'Intuitively, Assumption 4 is not sufficient to bound the regret ... because the agent can choose $\\pi$ to allocate treatment to single values of $\\tilde u\\in\\mathcal U$', corrected: Assumption 2, and the selected type is the one left untreated", "location": "PDF p. 23 (Appendix D.3, paragraph after Assumption 4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T17" }, { "declarations": [ { "claim_revision_id": "catalog:1f9572ad6c35b03176e4e745f6edef548df7c6e97e17bdcc0519581b760654a0", "conditional_on": [], "credit_reason": "Exact witness for the printed sign error; finite Boolean sums retain the zero-on-empty reward and all maxima are attained.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T18_regret_sign_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L590", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1f9572ad6c35b03176e4e745f6edef548df7c6e97e17bdcc0519581b760654a0", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T18", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T18" }, "html": { "body": "

The printed claim is false: P006-T18; PDF p. 17 (Appendix B.4, proof of Proposition 1). The fair two-unit deterministic example has samplewise ATO bounded by the event that some type is 1, with equality under (0,1). Its exact reward is b(4-a-b)/4, uniquely maximized at (0,1), with value 3/4. Welfare is a/2-3b/2, with maximum 1/2 and selected value -3/2. Regret is 2; the printed expression is -1. Every feasible policy's regret from an attained maximum is nonnegative.

", "headline": "Proof of Proposition 1 (ATO Regret), second failure mode: the display R(πw)=(1−p)−p(β−1)=1−pβR(\\pi^w)=(1-p)-p(\\beta-1)=1-p\\betaR(πw)=(1−p)−p(β−1)=1−pβ", "location": "PDF p. 17 (Appendix B.4, proof of Proposition 1)" }, "id": "P006-T18", "text": { "body": "The printed claim is false: P006-T18; PDF p. 17 (Appendix B.4, proof of Proposition 1). The fair two-unit deterministic example has samplewise ATO bounded by the event that some type is 1, with equality under (0,1). Its exact reward is b(4-a-b)/4, uniquely maximized at (0,1), with value 3/4. Welfare is a/2-3b/2, with maximum 1/2 and selected value -3/2. Regret is 2; the printed expression is -1. Every feasible policy's regret from an attained maximum is nonnegative.", "headline": "Proof of Proposition 1 (ATO Regret), second failure mode: the display $R(\\pi^w)=(1-p)-p(\\beta-1)=1-p\\beta$", "location": "PDF p. 17 (Appendix B.4, proof of Proposition 1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T18" }, { "declarations": [ { "claim_revision_id": "catalog:50c9f32760796eafd970e1a59ec84602de45baa5bc9387af8f153fac77b2c2a1", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'confirmed', 'what': \"P006/P006_CounterfactualMetrics.lean:608 states the finite-parameter optimum, welfare, regret and arithmetic comparisons in T19(i)-(iii), but contains no limit conclusion. Corrected T19(iv) and this row's final sentence expressly assert that regret tends to infinity as β tends to infinity. T19_corrected_regret_tendsto at line 630 supplies that result separately, so the omission is real for this credited row but does not leave the work's T19 limit unformalized.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T19_corrected_regret", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L608", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:50c9f32760796eafd970e1a59ec84602de45baa5bc9387af8f153fac77b2c2a1", "conditional_on": [], "credit_reason": "The larger p domain is harmless: only 0PDF p. 17 (Appendix B.4, proof of Proposition 1). The corrected second-mode welfare and regret formulas have a positive discrepancy from the printed formula and diverge as beta increases; the half-probability example is stated in its beta-greater-than-one range.

", "headline": "Proof of Proposition 1, second failure mode, corrected: R(πw)=(1−p)−p(1−β)=1−2p+pβR(\\pi^w)=(1-p)-p(1-\\beta)=1-2p+p\\betaR(πw)=(1−p)−p(1−β)=1−2p+pβ", "location": "PDF p. 17 (Appendix B.4, proof of Proposition 1)" }, "id": "P006-T19", "text": { "body": "PDF p. 17 (Appendix B.4, proof of Proposition 1). The corrected second-mode welfare and regret formulas have a positive discrepancy from the printed formula and diverge as beta increases; the half-probability example is stated in its beta-greater-than-one range.", "headline": "Proof of Proposition 1, second failure mode, corrected: $R(\\pi^w)=(1-p)-p(1-\\beta)=1-2p+p\\beta$", "location": "PDF p. 17 (Appendix B.4, proof of Proposition 1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T19" }, { "declarations": [ { "claim_revision_id": "catalog:b59724f783886b24a454aaef2326ad8f0ec860c7b2559a3af333715096f7519d", "conditional_on": [], "credit_reason": "The witness tests the mathematical coding formula, not whether a particular regression was fitted correctly; it uses the printed TO positive-treated-mean rule.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T20_logistic_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L953", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:b59724f783886b24a454aaef2326ad8f0ec860c7b2559a3af333715096f7519d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T20", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T20" }, "html": { "body": "

PDF p. 25 (Appendix E.2.1). The normalized four-state logistic witness has signed means zero and minus one half, contradicting the printed probability-as-mean formula and changing the TO decision.

", "headline": "Appendix E.2.1 (Horse Colic dataset), the formula 'μt(x)=σ(β^0+β^1⊤x+β^2t+β^3⊤xt)\\mu_t(x)=\\sigma(\\hat\\beta_0+\\hat\\beta_1^\\top x+\\hat\\beta_2t+\\hat\\beta_3^\\top xt)μt​(x)=σ(β^​0​+β^​1⊤​x+β^​2​t+β^​3⊤​xt)' for the conditional mean of an outcome coded 111 / −1-1−1", "location": "PDF p. 25 (Appendix E.2.1)" }, "id": "P006-T20", "text": { "body": "PDF p. 25 (Appendix E.2.1). The normalized four-state logistic witness has signed means zero and minus one half, contradicting the printed probability-as-mean formula and changing the TO decision.", "headline": "Appendix E.2.1 (Horse Colic dataset), the formula '$\\mu_t(x)=\\sigma(\\hat\\beta_0+\\hat\\beta_1^\\top x+\\hat\\beta_2t+\\hat\\beta_3^\\top xt)$' for the conditional mean of an outcome coded $1$ / $-1$", "location": "PDF p. 25 (Appendix E.2.1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T20" }, { "declarations": [ { "claim_revision_id": "catalog:995e1753c6c7e62408497d7de6171f6ea70ee027bfff6b432e4a6398b39eaa8f", "conditional_on": [], "credit_reason": "The sum defining signedBernoulliMean uses exactly the two possible signed outcomes, so this representation loses no distributional case for the asserted conditional mean. Empty covariate dimension is harmless.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T21_corrected_logistic_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L931", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:995e1753c6c7e62408497d7de6171f6ea70ee027bfff6b432e4a6398b39eaa8f", "conditional_on": [], "credit_reason": "Auxiliary fact for T21(iii); the positive exponential prevents zero denominators.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.logistic_bounds", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L916", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:995e1753c6c7e62408497d7de6171f6ea70ee027bfff6b432e4a6398b39eaa8f", "conditional_on": [], "credit_reason": "Auxiliary algebra for T21(i), with no extra model assumptions and no use of totalized probabilistic integrals.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.signed_mean_formula", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L922", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:995e1753c6c7e62408497d7de6171f6ea70ee027bfff6b432e4a6398b39eaa8f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T21", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T21" }, "html": { "body": "

PDF p. 25 (Appendix E.2.1). For outcomes coded as one and minus one, logistic probabilities give means 2 sigma minus 1, twice the probability difference as the effect, and never the printed mean.

", "headline": "Appendix E.2.1, the conditional-mean formula corrected to μt(x)=2σ(β^0+β^1⊤x+β^2t+β^3⊤xt)−1\\mu_t(x)=2\\sigma(\\hat\\beta_0+\\hat\\beta_1^\\top x+\\hat\\beta_2t+\\hat\\beta_3^\\top xt)-1μt​(x)=2σ(β^​0​+β^​1⊤​x+β^​2​t+β^​3⊤​xt)−1", "location": "PDF p. 25 (Appendix E.2.1)" }, "id": "P006-T21", "text": { "body": "PDF p. 25 (Appendix E.2.1). For outcomes coded as one and minus one, logistic probabilities give means 2 sigma minus 1, twice the probability difference as the effect, and never the printed mean.", "headline": "Appendix E.2.1, the conditional-mean formula corrected to $\\mu_t(x)=2\\sigma(\\hat\\beta_0+\\hat\\beta_1^\\top x+\\hat\\beta_2t+\\hat\\beta_3^\\top xt)-1$", "location": "PDF p. 25 (Appendix E.2.1)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T21" }, { "declarations": [ { "claim_revision_id": "catalog:f4ad1ad90d120748d0642702846485a59f45dc23f4b0f0dfb62c394bd8d830c9", "conditional_on": [], "credit_reason": "Decidable equality is only the representation needed to filter a finite sample and is available classically for every covariate type; it imposes no mathematical domain restriction. Nonempty-group and sample conditions are the conditions of the empirical formula.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T22_corrected_empirical_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1066", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f4ad1ad90d120748d0642702846485a59f45dc23f4b0f0dfb62c394bd8d830c9", "conditional_on": [], "credit_reason": "The target expressly evaluates the free-index denominator at a matching row, where its indicator is one; this is not a claim that the malformed printed expression defines an estimator for unmatched rows.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T22_two_row_example", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1130", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f4ad1ad90d120748d0642702846485a59f45dc23f4b0f0dfb62c394bd8d830c9", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T22", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T22" }, "html": { "body": "

PDF p. 26 (Appendix E.4). Nonempty empirical groups use the corrected count denominator, obey the bounds and zero-deviation identity, preserve the overall mean, and give 2 in the two-row example rather than 4.

", "headline": "Appendix E.4 (Calculating γmarg\\gamma_{\\mathrm{marg}}γmarg​), the empirical conditional mean μ0(x)\\mu_0(x)μ0​(x), with the denominator ∑i=1n1(Xi=x)\\sum_{i=1}^n\\mathbb 1(X_i=x)∑i=1n​1(Xi​=x) (printed as the single indicator 1(Xi=x)\\mathbb 1(X_i=x)1(Xi​=x))", "location": "PDF p. 26 (Appendix E.4)" }, "id": "P006-T22", "text": { "body": "PDF p. 26 (Appendix E.4). Nonempty empirical groups use the corrected count denominator, obey the bounds and zero-deviation identity, preserve the overall mean, and give 2 in the two-row example rather than 4.", "headline": "Appendix E.4 (Calculating $\\gamma_{\\mathrm{marg}}$), the empirical conditional mean $\\mu_0(x)$, with the denominator $\\sum_{i=1}^n\\mathbb 1(X_i=x)$ (printed as the single indicator $\\mathbb 1(X_i=x)$)", "location": "PDF p. 26 (Appendix E.4)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T22" }, { "declarations": [ { "claim_revision_id": "catalog:740c94c8435b6e5f1cd15336f44e620a55ab1621136845ed9922fb87f4d38a1d", "conditional_on": [], "credit_reason": "Both treatment ignorability and observed-untreated positivity hold in the countermodels. The observed record includes R and missing values, so equality of records is not obtained by withholding an observable variable.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T23_missing_outcome_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1677", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:740c94c8435b6e5f1cd15336f44e620a55ab1621136845ed9922fb87f4d38a1d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T23", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T23" }, "html": { "body": "

PDF p. 13 (Appendix A.1, item 2). Two complete missing-outcome laws satisfy treatment independence and observation positivity and share every observed record, but have different untreated means.

", "headline": "Appendix A.1, item 2 ('Untreated patients in the treatment population'): conditions (a) and (b) as sufficient conditions for identifying μ0(x)\\mu_0(x)μ0​(x) when outcomes are observed for a fraction of the untreated units", "location": "PDF p. 13 (Appendix A.1, item 2)" }, "id": "P006-T23", "text": { "body": "PDF p. 13 (Appendix A.1, item 2). Two complete missing-outcome laws satisfy treatment independence and observation positivity and share every observed record, but have different untreated means.", "headline": "Appendix A.1, item 2 ('Untreated patients in the treatment population'): conditions (a) and (b) as sufficient conditions for identifying $\\mu_0(x)$ when outcomes are observed for a fraction of the untreated units", "location": "PDF p. 13 (Appendix A.1, item 2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T23" }, { "declarations": [ { "claim_revision_id": "catalog:9a6ba0185480acde457d0982120238211f42e30ea947ab84cbabed7193ec655e", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P006/Wave1/Gap/GP006B1.lean:324 exports all three corrected T24 conclusions for an arbitrary conditional-stratum probability law: identification under representativeness, the mixture and equality criterion when q<1, and representativeness from observation independence under the normalized untreated law. Independence is imposed under the correct conditional law rather than under the unconditioned population. Identification and the independence implication retain q=1, while only the missing-group comparison uses q<1, exactly as in the target.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.T24_corrected_identification_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B1.lean#L319", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9a6ba0185480acde457d0982120238211f42e30ea947ab84cbabed7193ec655e", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T24_mixture_mean_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1722", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9a6ba0185480acde457d0982120238211f42e30ea947ab84cbabed7193ec655e", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T24_representative_identification", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2249", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:9a6ba0185480acde457d0982120238211f42e30ea947ab84cbabed7193ec655e", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.independent_untreated_mean", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2210", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:9a6ba0185480acde457d0982120238211f42e30ea947ab84cbabed7193ec655e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T24", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T24" }, "html": { "body": "

PDF p. 13 (Appendix A.1, item 2). Within any supplied covariate-stratum law, mean representativeness identifies the untreated mean, the mixture gives the equality criterion, and observation independence implies representativeness.

", "headline": "Appendix A.1, item 2, corrected: add that the observed untreated outcomes are representative in mean", "location": "PDF p. 13 (Appendix A.1, item 2)" }, "id": "P006-T24", "text": { "body": "PDF p. 13 (Appendix A.1, item 2). Within any supplied covariate-stratum law, mean representativeness identifies the untreated mean, the mixture gives the equality criterion, and observation independence implies representativeness.", "headline": "Appendix A.1, item 2, corrected: add that the observed untreated outcomes are representative in mean", "location": "PDF p. 13 (Appendix A.1, item 2)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T24" }, { "declarations": [ { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P006/P006_CounterfactualMetrics.lean:2179 quantifies over every positive n, alpha, and epsilon, and produces a beta in (0,alpha) with an actual TT best response and regret at least lowerBoundGamma(alpha)-epsilon. The finite reward, welfare, and absolute-deviation definitions represent the admissible constant-X, fair-binary-U experiment, with the full-sample reward connected at line 2532. Selecting the displayed bad best response is faithful to T25(vi)'s existential reading, which does not require the bound for all tied responses.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T25_existential_lower_bound", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1865", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': \"P006/P006_CounterfactualMetrics.lean:2148 proves T25's formulas and optimization conclusions only for lowerBoundReward and lowerBoundWelfare, whose definitions at lines 2075 and 2079 use a two-coordinate policy, fair binary U, deterministic potential outcomes and the zero estimator. Corrected T25(i)-(v) covers independent, possibly nonconstant X, policies π(X,U) summarized by p_u=E[π(X,u)], outcomes with the specified conditional means, and estimators satisfying the T8 moment condition. The statement contains no general experiment or identity transporting its reward, welfare and policies to those two coordinates. The concrete family can support the existential Proposition 5 headline, but does not establish all the broader conditional family clauses credited to this row.\", 'suggested_fix': 'Quantify over the general X,U experiment in T25 and prove that actual expected TT reward, welfare and heterogeneity reduce to the displayed formulas with coordinates p_u=E[π(X,u)]; state the best-response characterization for all admissible X-dependent policies.', 'second_review': 'P006/P006_CounterfactualMetrics.lean:2148 proves the formulas and optimization results for two-coordinate policies and the specific lowerBoundReward and lowerBoundWelfare objects defined at lines 2075 and 2079. Corrected T25(i)–(v) also covers nonconstant independent X, policies π(X,U) summarized by p_u = E[π(X,u)], potential outcomes with the specified conditional means, and estimators satisfying the T8 moment condition. The declaration exports no reduction from those general experiments to its deterministic, zero-estimator binary model, so the credited family clauses require a general expectation and policy-reduction statement.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.T25_printed_lower_bound_family", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1834", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P006/P006_CounterfactualMetrics.lean:2222 supplies a positive perturbation delta below alpha, establishes existence of a best response, and proves uniqueness and the stated regret formula and bound for every best response in that construction. Taking beta=eta is allowed by T25(vii)'s existential unique-response lower-bound construction. This row covers that construction; the unequal-parameter optimizer calculation is separately stated at line 2196, and the unperturbed printed family retains its ties.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T25_unique_response_variant", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1908", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P006/P006_CounterfactualMetrics.lean:2532 equates the full product-sum expectation finiteIIDTT with n times the one-unit lowerBoundReward. The definitions at lines 2522 and 2075 use the printed TT reward with a zero baseline and retain every feasible stochastic policy in the fair-binary construction. Constant X, deterministic potential outcomes, and the exact zero estimator are admissible choices for T25's existential distribution construction, and the additional algebraic cases exclude none of its required witnesses.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T25_exact_sample_reward", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2191", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.scale_best_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1733", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P006/P006_CounterfactualMetrics.lean:2088 computes the exact one-unit reward, welfare, and zero marginal untreated baseline in T25's construction. Setting eta=0 yields the printed formulas, and the perturbation is represented by the same expressions with eta positive. These are algebraic evaluations of the chosen witness, with no assumptions carrying the conclusions; the full-sample multiplier and absolute-deviation equality are separate statements at lines 2532 and 2100.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.lower_bound_formulas", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1774", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.lower_bound_gamma_eq", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1786", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.lower_bound_best_response_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1794", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "The attained maximum ensures bestValue has its usual meaning; no empty/unbounded-supremum convention is involved. Uniqueness is not claimed by this row.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.lower_bound_welfare_optimum", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1813", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "Both directions of the equivalence are exported; membership and existence of the maximizer are not assumed as hypotheses.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.lower_bound_perturbed_best_iff", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L1882", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.iid_additive_expectation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2150", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:019905ef150f3fa5428d3d3c2d6a9dd115d2ecd7830a9cff3591358bf56ae69a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T25", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T25" }, "html": { "body": "

PDF p. 7 (Section 5); PDF pp. 22-23 (Appendix D.2). For arbitrary public covariates independent of a fair binary private type, integrable noisy outcomes and every estimator satisfying corrected T8, the TT reward, full response set, welfare, regret and minimal marginal heterogeneity are the stated two-coordinate formulas. The displayed bad response attains regret alpha-beta/2. Taking beta=min(epsilon,alpha/2) gives the lower bound. For 0<beta,eta<alpha the perturbed family has a unique response up to null sets and regret alpha-(beta+eta)/2; taking both parameters to be that minimum gives the same lower bound. The deterministic zero-estimator family supplies actual finite-sample existential witnesses.

", "headline": "Proposition 5 (lower bound on regret under information asymmetry) and the reward display of its proof with the unmatched parenthesis removed", "location": "PDF p. 7 (Section 5); PDF pp. 22-23 (Appendix D.2, restatement and proof)" }, "id": "P006-T25", "text": { "body": "PDF p. 7 (Section 5); PDF pp. 22-23 (Appendix D.2). For arbitrary public covariates independent of a fair binary private type, integrable noisy outcomes and every estimator satisfying corrected T8, the TT reward, full response set, welfare, regret and minimal marginal heterogeneity are the stated two-coordinate formulas. The displayed bad response attains regret alpha-beta/2. Taking beta=min(epsilon,alpha/2) gives the lower bound. For 0The printed claim is false: with a discrete covariate law having positive mass at every point of its stated space, the joint universal ATO/ATT best-response assertion fails under the standing assumptions and fixed-query unbiasedness. The statement negates the conjunction without claiming separate failures of both rules; the shared unbiasedness premise also applies to ATO, although Proposition 6 does not require it. PDF p. 15 (Proposition 6 and proof); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3).

", "headline": "Proposition 6 (ATO Best Response) and Proposition 7 (ATT Best Response), as printed, with the same rules in Section 3", "location": "PDF p. 15 (Proposition 6 and proof); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3)" }, "id": "P006-T26", "text": { "body": "The printed claim is false: with a discrete covariate law having positive mass at every point of its stated space, the joint universal ATO/ATT best-response assertion fails under the standing assumptions and fixed-query unbiasedness. The statement negates the conjunction without claiming separate failures of both rules; the shared unbiasedness premise also applies to ATO, although Proposition 6 does not require it. PDF p. 15 (Proposition 6 and proof); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3).", "headline": "Proposition 6 (ATO Best Response) and Proposition 7 (ATT Best Response), as printed, with the same rules in Section 3", "location": "PDF p. 15 (Proposition 6 and proof); PDF p. 16 (Proposition 7 and proof); PDF pp. 4-5 (Section 3)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T26" }, { "declarations": [ { "claim_revision_id": "catalog:86a61f36dd9018b74ffea7d1b709f03890a1007766b18adf0d9815f432880dc6", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'AronowAudit/P006/Wave1/TTAlignment.lean:17 negates precisely the universal conjunction containing the printed TT reward identity, best-response alignment, zero regret, and positive-effect response. Refuting the conjunction requires failure of at least one conjunct, exactly as the row says; it does not require separate refutations of all four. The finite witness has fixed-query unbiasedness without unbiasedness at the random evaluation covariate, which the printed Assumption 1 permits; its integrals and welfare maxima are genuine.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.Wave1.printed_TT_alignment_refuted", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/TTAlignment.lean#L17", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:86a61f36dd9018b74ffea7d1b709f03890a1007766b18adf0d9815f432880dc6", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T27", "decision": "refuted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T27" }, "html": { "body": "

The printed claim is false: fixed-query unbiasedness does not imply the joint universal TT alignment, maximizer, zero-regret and positive-rule assertions; this negates the conjunction without claiming separate failure of every clause. PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof).

", "headline": "Assumption 1 with the inference drawn from it in the proof of Proposition 9; Proposition 9 (TT Best Response) and Proposition 3 (TT Regret), as printed", "location": "PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof)" }, "id": "P006-T27", "text": { "body": "The printed claim is false: fixed-query unbiasedness does not imply the joint universal TT alignment, maximizer, zero-regret and positive-rule assertions; this negates the conjunction without claiming separate failure of every clause. PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof).", "headline": "Assumption 1 with the inference drawn from it in the proof of Proposition 9; Proposition 9 (TT Best Response) and Proposition 3 (TT Regret), as printed", "location": "PDF p. 4 (Assumption 1); PDF p. 5 (Reward Function 3, Proposition 3); PDF p. 17 (Proposition 9 and proof); PDF p. 18 (Proposition 3 and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T27" }, { "declarations": [ { "claim_revision_id": "catalog:ad9a2d7a30197297725f0c91fe53e390be3c7408812f0a9e75adefa14b6e3b35", "conditional_on": [], "credit_reason": "The only credited row is wrong_credit: it refutes a universal implication for hypothetical numerical summaries, without connecting those summaries or interval premises to the actual fitted effects of either reported analysis. It establishes neither the two particular beneficiary means nor their refutation.", "credit_status": "wrong_credit", "fidelity_status_baseline": "wrong_credit", "lean_name": "AronowAudit.P006.ConfidenceFollowup02.T28_printed_beneficiary_report_refuted", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_ConfidenceCompletion.lean#L16", "status": "counterexample_proved", "support_scope": "withdrawn", "used_in_solution": null } ], "formalization": { "claim_revision_id": "catalog:ad9a2d7a30197297725f0c91fe53e390be3c7408812f0a9e75adefa14b6e3b35", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T28", "decision": "omitted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T28" }, "html": { "body": "

No current theorem is supplied for this target.

", "headline": "Section 6, the two sentences giving the average benefit among the units that would benefit (0.147 for the Horse Colic data, 0.025 for the Stroke Trial data), as printed", "location": "PDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset')" }, "id": "P006-T28", "text": { "body": "No current theorem is supplied for this target.", "headline": "Section 6, the two sentences giving the average benefit among the units that would benefit (0.147 for the Horse Colic data, 0.025 for the Stroke Trial data), as printed", "location": "PDF p. 7 (Section 6, 'Horse Colic dataset'); PDF pp. 7-8 (Section 6, 'International Stroke Trial dataset')" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T28" }, { "declarations": [ { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"AronowAudit/P006/P006_CounterfactualMetrics.lean:642 gives, for every M>0, a negative alpha, a genuine ATO best response, an attained welfare maximum, and regret greater than M. The target explicitly reads “may be arbitrarily large” as existence of an experiment and response for each threshold, so n=2 and p=1/2 provide valid witnesses for every required M. Boundedness of each individual two-point mean function does not conflict with the target's stated unbounded-family reading.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T29_ato_unbounded", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L642", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "conditional_on": [], "credit_reason": "{'source': 'adjudication+second review', 'adjudicator': 'P006/Wave1/Gap/GP006B2.lean:312 characterizes maximizers of finiteIIDAverage n p outcomeP1, not the actual expected ATO reward of an arbitrary experiment with those conditional means. P006/P006_CounterfactualMetrics.lean:2254 defines that finite objective by inserting the fixed outcome m(X_i) into wATO and summing only covariates and treatments. T29(i) permits any integrable potential outcomes with the stated conditional means, including Y1=X+Z with independent mean-zero noise Z, and the theorem exports no equality reducing those experiments to its finite objective. The n=1 maximizing face and n≥2 unique maximizer are correct for the modeled objective, but the claimed family coverage is narrower than the target.', 'suggested_fix': 'Add a general expectation-reduction statement identifying E.expectedATO with finiteIIDAverage for every binary-covariate experiment satisfying the standing assumptions and the stated conditional means, and export the all-sample-size best-response characterization for that actual reward.', 'second_review': 'P006/Wave1/Gap/GP006B2.lean:312 characterizes maximizers of finiteIIDAverage, whose definition at P006/P006_CounterfactualMetrics.lean:2254 inserts the deterministic treated value outcomeP1(X_i) into wATO. T29(i) explicitly covers potential outcomes with the stated conditional means, including integrable outcome noise, for every positive sample size. The exported statement lacks the equality connecting actual Experiment.expectedATO for those experiments to the finite objective, so its all-sample-size characterization needs that general reduction and an actual-reward conclusion.', 'severity': 'major'}", "credit_status": "narrower", "fidelity_status_baseline": "narrower", "lean_name": "AronowAudit.P006.Wave1.T29_family_all_sizes_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/Wave1/Gap/GP006B2.lean#L308", "status": "proved", "support_scope": "limited", "used_in_solution": true }, { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "conditional_on": [], "credit_reason": "The row does not claim the displayed policy is the only ATO response at n=1. Dependence only on the conditional means is exactly what the target's welfare identity specifies.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T29_first_mode_welfare", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2114", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "conditional_on": [], "credit_reason": "The separate welfare lemma identifies the expression as the selected policy's regret in the probability-valued parameter range; no sample-size asymptotic is substituted.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.T29_first_mode_tendsto", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L2137", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "conditional_on": [], "credit_reason": "Its component is already supplied by the selected composite theorem; no separate use of this auxiliary declaration is needed.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P006.rewardP1_best", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P006/P006_CounterfactualMetrics.lean#L569", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:5e69fa2dd9f817ff55b410c2d1a4d6e61385a9a75b14514a09f6c308926f5d24", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P006-T29", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P006-T29" }, "html": { "body": "

PDF p. 4 (Section 3); PDF p. 17 (Appendix B.4). ATO regret exceeds every positive threshold. For every iid integrable conditional-outcome law in the binary family, its exact expected reward is h_n(q)pb. For n>=2 the unique response is (0,1); for n=1 exactly b=1 is required. In the first failure mode welfare has maximum -alpha(1-p)+p, selected value p and regret -alpha(1-p), which diverges as alpha tends to minus infinity at fixed 0<p<1. The claim allows arbitrary outcome noise.

", "headline": "Proposition 1 (ATO Regret)", "location": "PDF p. 4 (Section 3); PDF p. 17 (Appendix B.4, restatement and proof)" }, "id": "P006-T29", "text": { "body": "PDF p. 4 (Section 3); PDF p. 17 (Appendix B.4). ATO regret exceeds every positive threshold. For every iid integrable conditional-outcome law in the binary family, its exact expected reward is h_n(q)pb. For n>=2 the unique response is (0,1); for n=1 exactly b=1 is required. In the first failure mode welfare has maximum -alpha(1-p)+p, selected value p and regret -alpha(1-p), which diverges as alpha tends to minus infinity at fixed 0PDF p. 6 (Section 4); PDF pp. 18-19 (Appendix C.2). Under the usual measure-theoretic conditions (policies are measurable maps into [0,1]; outcomes, conditional means and the treated estimator residual are integrable), with corrected T6 and otherwise unconstrained policies, each agent's attained maximum TT score is its sample size times its mean positive effect. For two agents with the common untreated mean of the multi-agent setting, identical covariate laws, pointwise effect dominance and a positive mean positive effect for the dominated agent, these positive-effect means are ordered; a reversed sample-size-weighted inequality violates both ranking properties, for every reference law with finite effect means. The ratio condition is equivalent because its denominator is positive. Identical unit effects with sample sizes 1 and 2 give actual best scores 1 and 2 and witness the existential failure of both properties.

", "headline": "Proposition 4 (wTTw_{\\mathrm{TT}}wTT​ used directly as a score violates both ranking properties)", "location": "PDF p. 6 (Section 4); PDF pp. 18-19 (Appendix C.2, restatement and proof)" }, "id": "P006-T30", "text": { "body": "PDF p. 6 (Section 4); PDF pp. 18-19 (Appendix C.2). Under the usual measure-theoretic conditions (policies are measurable maps into [0,1]; outcomes, conditional means and the treated estimator residual are integrable), with corrected T6 and otherwise unconstrained policies, each agent's attained maximum TT score is its sample size times its mean positive effect. For two agents with the common untreated mean of the multi-agent setting, identical covariate laws, pointwise effect dominance and a positive mean positive effect for the dominated agent, these positive-effect means are ordered; a reversed sample-size-weighted inequality violates both ranking properties, for every reference law with finite effect means. The ratio condition is equivalent because its denominator is positive. Identical unit effects with sample sizes 1 and 2 give actual best scores 1 and 2 and witness the existential failure of both properties.", "headline": "Proposition 4 ($w_{\\mathrm{TT}}$ used directly as a score violates both ranking properties)", "location": "PDF p. 6 (Section 4); PDF pp. 18-19 (Appendix C.2, restatement and proof)" }, "url": "https://pmaronow.github.io/llm-audit/works/P006/#P006-T30" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "partial", "targets": [ { "agreement": "agrees_in_part", "location": "Section 6, PDF p. 9", "method": "I recalculated the table with SciPy replacements for the released regression code. Stroke values match the stored notebook outputs, with some last-digit differences from the printed table. Horse entries agree with the stored outputs and are approximately recovered by a rounded-coefficient reconstruction, but refitting with another solver changes them because no finite logistic maximum-likelihood estimate exists.", "status": "recalculated", "target": "Table 1" }, { "agreement": "to_rounding", "location": "Section 6, PDF p. 8", "method": "I recalculated all individual and cumulative Stroke feature values with replacement regression software. They agree with the stored notebook outputs to numerical precision and with the printed plot within visual reading precision.", "status": "reproduced", "target": "Figure 1" }, { "agreement": "agrees_in_part", "location": "Section 6, PDF p. 7, Horse Colic paragraph", "method": "I recalculated the released analysis with another logistic solver: it gives 59 positive fitted effects rather than the released 62. The released counts agree with the stored notebook outputs and coefficient reconstruction; no finite logistic maximum-likelihood estimate exists. The printed 0.147 is a population-average positive effect, not the mean among beneficiaries (about 0.6044; P006-E009).", "status": "recalculated", "target": "Horse Colic benefit counts and average benefit" }, { "agreement": "agrees_in_part", "location": "Section 6, PDF pp. 7-8, International Stroke Trial paragraph", "method": "I recalculated the released fit with replacement least-squares software. The sample size, 1,360 positive and 5,904 negative fitted effects, treatment rate and population-average positive effect agree. The printed 0.025 is not the mean among beneficiaries, which is about 0.1345 (P006-E009).", "status": "recalculated", "target": "Stroke Trial benefit counts and average benefit" }, { "agreement": "agrees_in_part", "location": "Appendix E.5, PDF p. 27", "method": "I recalculated the table with SciPy replacements for the released regression code. Stroke values match the stored notebook outputs, with some last-digit differences from the printed table. Horse entries agree with the stored outputs and are approximately recovered by a rounded-coefficient reconstruction, but refitting with another solver changes them because no finite logistic maximum-likelihood estimate exists.", "status": "recalculated", "target": "Table 2" }, { "agreement": "agrees_in_part", "location": "Appendix E.5, PDF p. 27", "method": "I recalculated both panels with replacement regression software. The Stroke panel agrees within plot-reading precision. The Horse panel agrees with the stored notebook outputs, but a refit with another logistic solver changes some bars because no finite maximum-likelihood estimate exists.", "status": "recalculated", "target": "Figure 2" }, { "agreement": "agrees_in_part", "location": "Appendix E.5, PDF pp. 27-28", "method": "I compared the Horse plot with the stored notebook outputs and recalculated it with another logistic solver. The stored plot agrees, but the refit changes bars and feature order. Solver refits give joint values of 0.0256467830 and 0.0280176387. For the released fit, exact bounds from the rounded outputs place the joint value in [0.028006, 0.028055], which rounds to 0.028 rather than the printed 0.035 (P006-N009).", "status": "recalculated", "target": "Figure 3 and joint $\\gamma_{\\mathrm{marg}}$" } ] }, "summary": "The paper models quality metrics as a principal--agent problem: a regulator rewards a provider that chooses whom to treat. It shows that rewarding the average treated outcome can cause unbounded regret, and that rewarding the total treatment effect, with shared covariate information and an estimate unbiased at evaluated treated observations, has zero regret. 22 entries are classified as errors (1 Major, 1 Moderate, 20 Minor), and 2 are clarity notes. Four other possible errors were not confirmed, and a fifth could not be determined. The Major finding, \\fid{P006-E001}, is that the best responses stated for the average rewards do not hold in finite samples. In the fitted Horse Colic simulation, the recomputed best response to the average-effect reward keeps 48\\% of the achievable welfare, compared with 5\\% under the printed rule.", "what_was_checked": "I checked the following by following the proofs, by recalculation from the replication files, or against the pages. They held up, subject to the corrections cited.\n\\begin{itemize}\n\\item Lemma 1: $V(\\pi)=E[\\tau(X)\\pi(X)]$, which is maximized over unconstrained policies by $\\mathbf{1}(\\tau(x)>0)$.\n\\item Propositions 8 and 9: $E[w_{\\mathrm{TO}}]=nE[\\mu_1(X)\\pi(X)]$ and $E[w_{\\mathrm{TT}}]=nV(\\pi)$ hold exactly for finite $n$, the second under the corrected Assumption 1 of \\fid{P006-E002}. Proposition 3 follows for any feasible set.\n\\item Proposition 1: the first example, with regret $-\\alpha(1-p)$, is correct; the second is correct after the sign correction of \\fid{P006-E007}.\n\\item Proposition 2: the bound $R\\le\\max_\\pi V(\\pi)$ holds for the finite-sample reward by a replacement proof (\\fid{P006-E001}).\n\\item Proposition 4: the example in which a larger population reverses the ranking is valid.\n\\item Theorem 1: under the corrected Assumption 1 and weighted integrability conditions of \\fid{P006-E002}, $E[w^g_{\\mathrm{TT}}]=nE[g(X)\\tau(X)\\pi(X)]$, and a strictly positive $g$ does not change the best response over unconstrained policies.\n\\item Theorem 2, with the corrections of \\fid{P006-E002}, \\fid{P006-E005} and \\fid{P006-E013}. Nonnegative treatment effects are needed for Definition 2.\n\\item Lemma 2, Proposition 10 and Theorem 3 for an estimator from independent auxiliary data. The bound $2\\gamma_{\\mathrm{marg}}$ was also tested on 6561 four-cell distributions in each of three policy classes and on 200000 random discrete instances, with no violation.\n\\item Proposition 5: the arithmetic of the construction, $R=\\alpha-\\beta/2$ and $\\gamma_{\\mathrm{marg}}=\\alpha$, for the displayed policy (\\fid{P006-N006}).\n\\item Lemma 3 and the inequalities in the proof of Theorem 5, given the positivity condition of \\fid{P006-E006}.\n\\item Stroke data: $n=7264$, treatment rate 0.3337, $Y\\in[-4,3]$, 17 categorical and 3 numerical covariates, 1360 and 5904 patients with positive and negative fitted effects. Every Stroke entry of Tables 1 and 2, the values plotted in Figure 1 and the Stroke panel of Figure 2 reproduce from the data, up to the last printed digits (\\fid{P006-E018}).\n\\item Horse Colic data: the preprocessing agrees with Appendix E.2.1 (20 features, 255 cases after exclusions, treatment rate $153/255=0.6$). The entries of Tables 1 and 2 and Figures 2 and 3 agree with the stored outputs of the released notebook, and a model reconstructed from the rounded coefficients in the released notebook reproduces the AUC 0.9924.\n\\item In the released calculations for Tables 1 and 2, before rounding, the regret is the $w_{\\mathrm{TT}}$ utility minus the utility of the row, and the treatment rates are whole numbers of units divided by 255 and by 7264.\n\\item Regret amplification in the Stroke data: regret 0.0891 with demographic information against 0.0741 with none. The comparison survives the corrections of \\fid{P006-E011} and \\fid{P006-N003}.\n\\item The released code implements the policies that the paper states and computes $\\gamma_{\\mathrm{marg}}$ as in Appendix E.4 with the denominator corrected in \\fid{P006-E015}. Whether the stated responses to the average rewards are best responses is the subject of \\fid{P006-E001}.\n\\end{itemize}\n\nThe following were not checked. Neither notebook was run in its original software environment, whose versions are not recorded. The Stroke least-squares fit was reproduced with other software. I did not rerun the Horse Colic logistic fit in its original software environment. I emulated it with other solvers and reconstructed it from the stored rounded coefficients. The reconstruction reproduces the stored AUC, while other solver fits change the results because no finite maximum-likelihood estimate exists. Of the cited works I read Dranove et al.\\ (2003) and the 2018 preprint of Kallus and Zhou. The other cited works were not read, and the paper's quotations were not compared with their sources. The clinical validity of the fitted potential outcomes was not assessed, and the paper itself says that it cannot be verified from the data.\n\nTwo files of the Stroke database, the trial protocol and a text version of the data, were not read. The data file in CSV form was read in full. I could not find the variables that the implementation of Kallus and Zhou used, and whether the coding of the outcome score for the Stroke data reproduces theirs is not settled. These checks do not show that the paper is correct in the respects that were not examined.", "work": { "arxiv_id": "2305.14595", "audit_complete": true, "authors": [ "Serena Wang", "Stephen Bates", "P. M. Aronow", "Michael I. Jordan" ], "category": "Articles - Theory and Methodology", "citation": "Serena Wang, Stephen Bates, P. M. Aronow, and Michael I. Jordan (2024). On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry. AISTATS 2024 PMLR 238:1522–1530.", "doi": null, "edition": { "pages_in_audited_file": 28, "sha256": "23f6cc84d5aec6abc601f4638b7981974c8cc4221ebcc7b9839309aba043627b", "source_url": "https://proceedings.mlr.press/v238/wang24b.html", "venue": "AISTATS 2024", "year": "2024" }, "id": "P006", "later_materials": null, "title": "On Counterfactual Metrics for Social Welfare: Incentives, Ranking, and Information Asymmetry", "type": "Conference paper", "url": "https://pmaronow.github.io/llm-audit/works/P006/", "venue": "AISTATS 2024", "year": "2024" } }