{ "claims": [ { "claim": "Under Monotonicity, No Liars and No Design Effects, Treatment Independence and non-degeneracy, prevalence is identified as $\\mu=E[Y_i]+E[1-Y_i](E[V_i\\mid Z_i=1,Y_i=0]-E[V_i\\mid Z_i=0,Y_i=0])$: the direct question identifies the contribution of those who admit the behaviour, and the list difference among those who answer 'No' identifies the remainder. Without Monotonicity the expression exceeds $\\mu$.", "claim_id": "P019-H1", "related_corrections": [ "P019-E008" ], "tied_to": "Lemma 1, equation (1), and the remark after its proof (PDF pp. 5-6, printed pp. 47-48); abstract and introduction (PDF pp. 1-3)" }, { "claim": "The combined estimator $\\hat\\mu=\\bar Y+(1-\\bar Y)(\\bar V_{1,0}-\\bar V_{0,0})$ is root-$n$ consistent and asymptotically normal with asymptotic variance (3), and the plug-in variance estimator supports Wald confidence intervals.", "claim_id": "P019-H2", "related_corrections": [ "P019-E002", "P019-E003", "P019-E008" ], "tied_to": "Lemma 2, Proposition 1 with equation (3), Corollary 1 (PDF pp. 6-7, printed pp. 48-49); Appendix B proofs (PDF pp. 20-21, printed pp. 62-63); abstract" }, { "claim": "Under the four assumptions the combined estimator is asymptotically strictly more efficient than the standard difference-in-means list estimator.", "claim_id": "P019-H3", "related_corrections": [ "P019-E008" ], "tied_to": "Proposition 2 (PDF p. 7, printed p. 49); proof in Appendix B, ending at (10) (PDF pp. 22-23, printed pp. 64-65); abstract" }, { "claim": "Direct answers permit two nonparametric placebo tests: among direct-question 'Yes' respondents the list difference in means equals 1 under the identifying assumptions (Placebo Test I, $\\beta=1$), and the list treatment has no effect on the direct answer under Treatment Independence (Placebo Test II, $\\delta=0$).", "claim_id": "P019-H4", "related_corrections": [ "P019-E001", "P019-E009", "P019-N001", "P019-N006" ], "tied_to": "Propositions 3 and 4 (PDF pp. 8-9, printed pp. 50-51); Tables 4, 5, 7, 8; supplement Tables A3, A4, A6, A7; abstract and introduction" }, { "claim": "In the Mechanical Turk application the combined estimator is much more precise than the standard list estimator: estimated sampling variance falls by 14 to 67 percent in Study A and by a similar range in Study B, with larger reductions where more respondents answer 'Yes'.", "claim_id": "P019-H5", "related_corrections": [ "P019-E007", "P019-E010" ], "tied_to": "Tables 3 and 6; Figures 1 and 2; Sections 5.2-5.3 (PDF pp. 12-15, printed pp. 54-57); discussion (PDF p. 16, printed p. 58)" }, { "claim": "Placebo Test I rejects in two of five cases in Study A and in none in Study B (a Fisher joint test is insignificant in Study B); in the replication it rejects in one of five cases in Study A and three of five in Study B: six rejections in twenty opportunities, read as many instances of violated assumptions.", "claim_id": "P019-H6", "related_corrections": [ "P019-N003" ], "tied_to": "Tables 4 and 7; supplement Tables A3 and A6; Sections 5.2-5.4 (PDF pp. 12-16, printed pp. 54-58); discussion (PDF p. 17, printed p. 59)" }, { "claim": "Question order matters: the standard list estimates differ between Study A and Study B while the direct and combined estimates do not, and asking the direct question after the list affects direct answers (nuclear power and CNN; the CNN effect replicates). Hence the recommendation to ask direct questions where feasible and to randomize the order of direct and list questions.", "claim_id": "P019-H7", "related_corrections": [ "P019-E009" ], "tied_to": "Table 8; supplement Tables A1, A7 and A8 and Appendices E and F; Section 5.3 (PDF pp. 14-15), Section 5.4 (PDF p. 16), discussion (PDF pp. 16-17, printed pp. 58-59)" }, { "claim": "Placebo Test I has modest power: about 30 percent when 20 percent of 200 'Yes' respondents are false confessors and about 80 percent with 800; the supplement adds that 80 percent power is reached at 800 when about 20 percent lie on the treatment list or are design affected.", "claim_id": "P019-H8", "related_corrections": [ "P019-E004", "P019-E005", "P019-E006" ], "tied_to": "Section 4.1, last paragraph (PDF pp. 8-9, printed pp. 50-51); supplement Appendix C and Figure A1 (supplement PDF pp. 2-3); for Placebo Test II, Appendix D and Figure A2 (supplement PDF pp. 3-4); discussion (PDF p. 17, printed p. 59)" }, { "claim": "A replication with 1,012 new subjects gave findings strikingly similar to the first investigation; the list experimental estimates vary somewhat between the original experiments and the replication, but none of the differences is significant.", "claim_id": "P019-H9", "related_corrections": [], "tied_to": "Section 5.4 (PDF pp. 15-16, printed pp. 57-58); footnote 1 (PDF p. 12, printed p. 54); supplement Appendix F and Tables A2-A8 against Tables 3-8" } ], "date_published": "2026-10-25", "findings": [ { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of Proposition 1 with equation (3) and the four lines that follow. A red box marks 'Assumption 4 (Non-Degenerate Distributions),' in 'Given Assumption 4 (Non-Degenerate Distributions), the asymptotic variance of mu-hat is characterized by'. The paragraph after the equation begins 'The proof is given in Appendix B. Under Assumptions 1-4, mu-hat is root-n consistent and asymptotically normal'.", "caption": "Proposition 1, p. 49 (PDF p. 7). The marked hypothesis names Assumption 4 alone, and the sentence after the proposition says “Under Assumptions 1–4”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E002.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T18", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T18" } ], "headline": "Proposition 1 names only Assumption 4, but its variance formula (3) also uses Assumptions 1–3; the hypothesis should read “Given Assumptions 1–4”.", "id": "P019-E002", "location": "Proposition 1, p. 49 (PDF p. 7)", "pdf_page": 7, "printed_page": "49", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P019-T18" ], "text": { "as_printed": "``Proposition 1. Given \\hl{Assumption 4 (Non-Degenerate Distributions)}, the asymptotic variance of $\\hat\\mu$ is characterized by''\n\\[\n\\begin{aligned}\n\\operatorname*{plim}_{n\\to\\infty} n\\operatorname{Var}[\\hat\\mu]={}&\\frac{(1-\\mu)^2}{1-\\mu p}\\,\\mu p+(1-\\mu p)\\\\\n&\\times\\left[\\frac{\\operatorname{Var}[V_i\\mid Z_i=1,Y_i=0]}{\\gamma}+\\frac{\\operatorname{Var}[V_i\\mid Z_i=0,Y_i=0]}{1-\\gamma}\\right].\\qquad(3)\n\\end{aligned}\n\\]", "corrected": "``Proposition 1. Given \\hl{Assumptions 1--4}, the asymptotic variance of $\\hat\\mu$ is characterized by''\n\\[\n\\operatorname*{plim}_{n\\to\\infty} n\\operatorname{Var}[\\hat\\mu]=\\frac{(1-\\mu)^2}{1-\\mu p}\\,\\mu p+(1-\\mu p)\\left[\\frac{\\operatorname{Var}[V_i\\mid Z_i=1,Y_i=0]}{\\gamma}+\\frac{\\operatorname{Var}[V_i\\mid Z_i=0,Y_i=0]}{1-\\gamma}\\right].\\qquad(3)\n\\]", "intended_meaning": "The proposition is stated within the model of Section 2, in which Assumptions 1--3 are maintained, and Assumption 4 is the condition added for inference. The sentence after the proposition, Corollary 1 and Proposition 2 all say ``Assumptions 1--4'', and under these four assumptions formula (3) is correct. Read as a self-contained statement, the proposition omits hypotheses that (3) needs. To see where they enter, I write $q=\\mathrm{E}[Y_i]$, $m_z=\\mathrm{E}[V_i\\mid Z_i=z,Y_i=0]$, $d=m_1-m_0$ and $s_z=\\operatorname{Var}[V_i\\mid Z_i=z,Y_i=0]$. For independent observations with $Z_i$ independent of $Y_i$, the delta method gives the asymptotic variance\n\\[\nq(1-q)(1-d)^2+(1-q)\\left[\\frac{s_1}{\\gamma}+\\frac{s_0}{1-\\gamma}\\right].\n\\]\nBy Assumption 1, I have $q=\\mu p$, and Assumptions 1--3 imply that $d=\\mathrm{E}[X_i\\mid Y_i=0]=(\\mu-\\mu p)/(1-\\mu p)$, so that $1-d=(1-\\mu)/(1-\\mu p)$. Substituting these values, I obtain (3). Note that the formula can fail without Assumption 1. Let $X_i$, $Y_i$, $W_i$ and $Z_i$ be independent Bernoulli variables with mean $1/2$, and set $V_i=W_i+X_iZ_i$. Then Assumptions 2--4 hold and Assumption 1 does not. In this example $\\mu=p=\\gamma=1/2$, $s_1=1/2$ and $s_0=1/4$, the asymptotic variance of $\\hat\\mu$ is $13/16$, and the right side of (3) is $29/24$. The proof of the proposition in Appendix B is the subject of \\fid{P019-E003}." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E002", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of display (6) in the proof of Lemma 2, with the two lines of text before it and the line after it. A red box marks the sum over i from 1 to k of the probability that the sum of Z-j over j from 1 to k equals i. The display sets the conditional expectation of V-bar-1,0 given 1 minus Y-bar equal to k over n first to E of V-1 given Z-1 equal to 1 and Y-1 equal to 0 times this sum, and then to that expectation alone.", "caption": "Proof of Lemma 2, display (6), p. 62 (PDF p. 20). The marked sum is set equal to 1 in the next line; it equals $1-(1-\\gamma)^{k}$, because the case of no treated subject among the $k$ is left out.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003.png" }, { "alt": "Crop of the proof of Proposition 1. Its first line reads 'Proof. The proof proceeds by working with linearized variances:' and is followed by display (8), which writes the variance of mu-hat as the variance of the product of 1 minus Y-bar and the quantity V-bar-1,0 minus V-bar-0,0 minus 1. Red boxes mark the words 'are uncorrelated,' in the sentence 'By Lemma 2, V-bar-1,0 and V-bar-0,0 are uncorrelated, so the variance of the product in (8) decomposes as follows:' and the first equality of (9), which writes the variance of mu-hat as a sum of three products of means and variances of 1 minus Y-bar and of V-bar-1,0 minus V-bar-0,0. The next equality in the crop ends with plus O of n to the minus 2.", "caption": "Proof of Proposition 1, from its first line through the first two equalities of display (9), p. 63 (PDF p. 21). The proof begins by saying that it works with linearized variances. The marked equality is the variance formula for a product of independent variables; it is printed without a remainder term, and the next equality carries $+O(n^{-2})$. The marked words omit “with $1-\\bar Y$”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-b.png" }, { "alt": "Crop of the last line of display (9) and the sentence after it. A red box marks the equation n gamma equals the sum of Z-i over i from 1 to n, in the sentence 'where gamma is the probability of receiving treatment, so n gamma equals the sum of Z-i. Multiplying by n yields the desired result.'", "caption": "Proof of Proposition 1, sentence after display (9), p. 63 (PDF p. 21). The marked equation sets $n\\gamma$, an expected count, equal to the realized number of treated subjects.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-c.png" }, { "alt": "Crop of the definitions of the sample means in Section 3 and of Lemma 2. Red boxes mark the two ratios that define V-bar-1,0 and V-bar-0,0, each a sum of list responses divided by the number of subjects in the subgroup, and the statement 'Lemma 2. V-bar-1,0 and V-bar-0,0 are uncorrelated with 1 minus Y-bar.'", "caption": "Section 3, p. 48 (PDF p. 6). The two marked ratios are $0/0$ when their subgroup is empty, and the marked lemma is stated without a condition.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E003-d.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T3", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T3" }, { "id": "P019-T4", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T4" }, { "id": "P019-T18", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T18" } ], "headline": "The proofs of Lemma 2 and Proposition 1 in Appendix B are worded as exact arguments for a finite sample, but they are meant to leading order and on samples in which both subgroups of subjects who answer “No” are non-empty, as the announced linearization indicates; read in this way they are correct, and formula (3) is unchanged.", "id": "P019-E003", "location": "Appendix B.1, proof of Lemma 2, displays (5)–(7), pp. 62–63 (PDF pp. 20–21); proof of Proposition 1, display (9) and the sentences before and after it, p. 63 (PDF p. 21); definitions of the subgroup means and Lemma 2, Section 3, p. 48 (PDF p. 6)", "pdf_page": 20, "printed_page": "62–63", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [ "P019-T18", "P019-T3", "P019-T4" ], "text": { "as_printed": "Section 3, p.~48:\n\\[\n\\bar V_{1,0}=\\frac{\\sum_{i=1}^{n}V_iZ_i(1-Y_i)}{\\sum_{i=1}^{n}Z_i(1-Y_i)}\n\\qquad\\text{and}\\qquad\n\\bar V_{0,0}=\\frac{\\sum_{i=1}^{n}V_i(1-Z_i)(1-Y_i)}{\\sum_{i=1}^{n}(1-Z_i)(1-Y_i)}.\n\\]\n``Lemma 2. $\\bar V_{1,0}$ and $\\bar V_{0,0}$ \\hl{are uncorrelated with} $1-\\bar Y$.''\n\nProof of Lemma 2, p.~62, display (6):\n\\[\n\\mathrm{E}\\Bigl[\\bar V_{1,0}\\Bigm|1-\\bar Y=\\frac kn\\Bigr]\n=\\mathrm{E}[V_1\\mid Z_1=1,Y_1=0]\\,\\hl{\\sum_{i=1}^{k}\\Pr\\Bigl(\\sum_{j=1}^{k}Z_j=i\\Bigr)}\n=\\mathrm{E}[V_1\\mid Z_1=1,Y_1=0].\n\\]\nDisplay (7), p.~63, has the same step for $\\bar V_{0,0}$, with $\\Pr\\bigl(\\sum_{j=1}^{k}(1-Z_j)=i\\bigr)$.\n\nProof of Proposition 1, p.~63. The proof begins ``The proof proceeds by working with linearized variances:'', followed by display (8). It continues: ``By Lemma 2, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ \\hl{are uncorrelated}, so the variance of the product in (8) decomposes as follows:''\n\\[\n\\begin{aligned}\n\\operatorname{Var}[\\hat\\mu]\\ \\hl{=}\\ {}&(\\mathrm{E}[1-\\bar Y])^{2}\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}]\n+\\operatorname{Var}[1-\\bar Y]\\,(\\mathrm{E}[\\bar V_{1,0}-\\bar V_{0,0}-1])^{2}\\\\\n&+\\operatorname{Var}[1-\\bar Y]\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}]\n\\end{aligned}\n\\]\nThis is the first equality of (9); the later lines of (9) carry $+O(n^{-2})$.\n\nAfter (9), p.~63: ``where $\\gamma$ is the probability of receiving treatment, so $\\hl{n\\gamma=\\sum_{i=1}^{n}Z_i}$. Multiplying by $n$ yields the desired result.''", "corrected": "\\emph{Convention.} When a subgroup is empty, its mean is set to a fixed bounded value, for example 0, and the variance estimator is set to a fixed bounded value wherever its formula is undefined. This is a definition and not a new assumption on the population.\n\n\\emph{Lemma 2.} Suppose that $\\hl{n\\ge2}$, that the subjects are independent and identically distributed and that Assumptions 1--4 hold. Let $N_{z,0}=\\sum_{i=1}^{n}\\mathbf 1\\{Z_i=z,\\,Y_i=0\\}$ for $z\\in\\{0,1\\}$ and $D=\\{N_{1,0}>0,\\ N_{0,0}>0\\}$. Then, on $D$,\n\\[\n\\mathrm{E}[\\bar V_{z,0}\\mid Y_1,Z_1,\\dots,Y_n,Z_n]=\\mathrm{E}[V_i\\mid Z_i=z,Y_i=0]\\qquad(z=0,1),\n\\]\nand, \\hl{conditionally on $D$}, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated with $1-\\bar Y$. In the proof, condition on all of $(Y_1,Z_1,\\dots,Y_n,Z_n)$ in place of the argument of displays (4)--(7).\n\n\\emph{Proof of Proposition 1.} ``By Lemma 2, $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated \\hl{with $1-\\bar Y$ given $D$}. \\hl{Using conditional total variance and a first-order expansion}, the variance of the product in (8) decomposes as follows:''\n\\[\n\\begin{aligned}\n\\operatorname{Var}[\\hat\\mu\\mid D]={}&(\\mathrm{E}[1-\\bar Y\\mid D])^{2}\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}\\mid D]\n+\\operatorname{Var}[1-\\bar Y\\mid D]\\,(\\mathrm{E}[\\bar V_{1,0}-\\bar V_{0,0}-1\\mid D])^{2}\\\\\n&+\\operatorname{Var}[1-\\bar Y\\mid D]\\operatorname{Var}[\\bar V_{1,0}-\\bar V_{0,0}\\mid D]\\ \\hl{{}+O(n^{-2})}.\n\\end{aligned}\n\\]\nThe remaining lines of (9) hold to their stated order. With the convention above, $n\\operatorname{Var}[\\hat\\mu]$ converges to the right-hand side of (3), whichever bounded values are chosen.\n\nAfter (9): ``where $\\gamma$ is the probability of receiving treatment, so $n\\gamma=\\hl{\\mathrm{E}}\\bigl[\\sum_{i=1}^{n}Z_i\\bigr]$. Multiplying by $n$ yields the desired result.''", "intended_meaning": "The proof of Proposition 1 begins with the words ``The proof proceeds by working with linearized variances'', the later lines of (9) carry $+O(n^{-2})$, and the proposition states a limit. The argument of Appendix B is therefore meant to leading order in $n$. The subgroup means are ratios, which are defined only when both subgroups of subjects with $Y_i=0$ are non-empty, and the argument concerns such samples. On this reading it is correct, although its steps are worded as exact statements for a finite sample.\n\nLet $N_{z,0}$ be the number of subjects with $Z_i=z$ and $Y_i=0$, and let $D=\\{N_{1,0}>0,\\ N_{0,0}>0\\}$. I write $m_z$ and $s_z$ for the mean and the variance of $V_i$ given $Z_i=z$ and $Y_i=0$. Under independent sampling, the complement of $D$ has positive probability for every $n$. The sums in (6) and (7) leave out the case of an empty subgroup, and they equal $1-(1-\\gamma)^{k}$ and $1-\\gamma^{k}$, not 1. Given all the direct answers and assignments, on $D$ the mean $\\bar V_{z,0}$ has conditional expectation $m_z$ and conditional variance $s_z/N_{z,0}$. Since $1-\\bar Y$ is determined by the direct answers, this implies that $\\bar V_{1,0}$ and $\\bar V_{0,0}$ are uncorrelated with $1-\\bar Y$ conditionally on $D$. This is the lemma as intended.\n\nThe first equality of (9) is the formula for the variance of a product of two independent variables, and Lemma 2 does not imply it as an exact identity. Let $h=s_1/N_{1,0}+s_0/N_{0,0}$, which on $D$ is the conditional variance of $\\bar V_{1,0}-\\bar V_{0,0}$ given all the direct answers and assignments. By the law of total variance, I find that $\\operatorname{Var}[\\hat\\mu\\mid D]$ minus the right-hand side of the first equality, with its moments taken conditionally on $D$, equals\n\\[\n\\operatorname{Cov}\\bigl((1-\\bar Y)^{2},h\\bigm|D\\bigr)=-\\frac{2\\mu p}{n^{2}}\\Bigl(\\frac{s_1}{\\gamma}+\\frac{s_0}{1-\\gamma}\\Bigr)+o(n^{-2}).\n\\]\nThis term has the order of the remainder in the later lines of (9), and the limit in (3) does not depend on it. The sentence before (9) cites Lemma 2 for the statement that the two subgroup means are uncorrelated. The lemma says that each is uncorrelated with $1-\\bar Y$, and the words ``with $1-\\bar Y$'' are to be supplied.\n\nSince $\\gamma=\\Pr(Z_i=1)$ is a population quantity, $n\\gamma$ is the expected number of treated subjects, which is what the calculation to leading order uses. The realized number is $\\sum_iZ_i=n\\hat\\gamma$, with $\\hat\\gamma$ as defined on p.~49. In the first list experiment of Study A, 245 of the 500 subjects were treated, while $n\\gamma=250$.\n\nFormula (3), the asymptotic normality of $\\hat\\mu$, Corollary 1 and Proposition 2 are unaffected. Formula (3) also follows from the influence function of $\\hat\\mu$, without Lemma 2. In the original main analysis every subgroup defined by $(Z_i,Y_i)$ has at least 21 subjects. The hypothesis of Proposition 1 is the subject of \\fid{P019-E002}." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E003", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the first paragraph of Section 4.1 and three lines of the next. A red box marks 'V_i + 1' in the sentence 'Any valid test of distributional equality between V_i (under Z_i = 0, Y_i = 1) and V_i + 1 (under Z_i = 1, Y_i = 1) will permit rejection of the null.' The sentence before it states Pr[V_i = t | Z_i = 0, Y_i = 1] = Pr[V_i = (t + 1) | Z_i = 1, Y_i = 1].", "caption": "Section 4.1, first paragraph, p. 50 (PDF p. 8). The marked term shifts the list response of the treated arm up by one, and the identity two lines above it requires a shift down.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E001.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T5", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T5" }, { "id": "P019-T6", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T6" } ], "headline": "The sentence on distributional tests shifts the list response of the treated arm the wrong way: the response to compare with $V_i$ in the control arm is $V_i-1$ in the treated arm, not $V_i+1$.", "id": "P019-E001", "location": "Section 4.1, first paragraph, p. 50 (PDF p. 8)", "pdf_page": 8, "printed_page": "50", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T5", "P019-T6" ], "text": { "as_printed": "``Under these assumptions, for all $t$, $\\Pr[V_i=t\\mid Z_i=0,Y_i=1]=\\Pr[V_i=(t+1)\\mid Z_i=1,Y_i=1]$, thus tests of distributional equality are appropriate. Any valid test of distributional equality between $V_i$ (under $Z_i=0$, $Y_i=1$) and $\\hl{V_i+1}$ (under $Z_i=1$, $Y_i=1$) will permit rejection of the null.''", "corrected": "``Any valid test of distributional equality between $V_i$ (under $Z_i=0$, $Y_i=1$) and $\\hl{V_i-1}$ (under $Z_i=1$, $Y_i=1$) will permit rejection of the null.''\n\nIn full: suppose that Assumptions 1--3 hold and that the events $\\{Z_i=z,Y_i=1\\}$, $z\\in\\{0,1\\}$, have positive probability, as Assumption 4 requires. Then for all $t$,\n\\[\n\\Pr[V_i=t\\mid Z_i=0,Y_i=1]=\\Pr[V_i-1=t\\mid Z_i=1,Y_i=1].\n\\]\nEquivalently, one may compare $V_i+1$ in the control arm with $V_i$ in the treated arm. The scalar test of $\\beta=1$ remains valid.", "evidence": "The evidence is a derivation and an exact counterexample to the printed comparison. Suppose that Assumptions 1--3 hold and that $Y_i=1$. Then $X_i=1$ by Assumption 1, and by Assumption 2 I have $V_i=W_i$ in the control arm and $V_i=W_i+1$ in the treated arm. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms. I conclude that $V_i$ in the control arm and $V_i-1$ in the treated arm have the same distribution. For the counterexample, let $X_i=Y_i$ be a Bernoulli variable with mean $1/2$, let $W_i$ and $Z_i$ be Bernoulli variables with mean $1/2$ such that $X_i$, $W_i$ and $Z_i$ are independent, and set $V_i=W_i+X_iZ_i$. All four assumptions hold. Among respondents with $Y_i=1$, the response $V_i$ in the control arm takes the values $0$ and $1$ with probability $1/2$ each. In the treated arm, $V_i+1$ takes the values $2$ and $3$ with probability $1/2$ each, and $V_i-1$ takes the values $0$ and $1$ with probability $1/2$ each.", "problem": "The identity in the first sentence says that, among respondents who answer ``Yes'', the list response in the treated arm is distributed as the list response in the control arm plus one. The distributions that are equal under the null are therefore those of $V_i$ in the control arm and $V_i-1$ in the treated arm, or equivalently those of $V_i+1$ in the control arm and $V_i$ in the treated arm. The second sentence compares $V_i$ in the control arm with $V_i+1$ in the treated arm, that is, $W_i$ with $W_i+2$. These two distributions need not be equal when all four assumptions hold, and a test built on the sentence as printed can systematically reject a valid model.", "what_survives": "The probability identity, the null $\\beta=1$, the test of Proposition 3 and the reported placebo tests are unaffected. The replication code tests the difference in means against 1, as the paper describes. Only a distributional test implemented from the sentence as printed would use the wrong comparison." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E001", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of seven lines of Appendix C. Red boxes mark the sentence 'The treatment list response for Design Affected subjects was generated as a ceiling effect: the control list plus one, except for those with a 4 on the control list; those units' treatment list response remained equal to 4.', which ends with the footnote mark 1. The crop begins partway through the sentence specifying a binomial distribution with a success probability of 0.4 and four trials. The lines above the marked sentence say that treatment list responses of units meeting the assumptions were set equal to the control list response plus one.", "caption": "Online supplement, Appendix C, second paragraph, p. 2. The marked sentence describes the simulated design effect as the control-list response plus one, capped at 4. The superscript refers to footnote 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004.png" }, { "alt": "Crop of four lines. Red boxes mark the words 'or change their responses to the non-sensitive items on the treatment list, the placebo test achieves 80 percent power.' in the sentence that begins 'When approximately 20 percent of a sample of 800 confessors falsely confess, lie on the treatment list,'. The page prints percent signs.", "caption": "Online supplement, first lines of p. 3. The marked clause extends the statement of 80% power to the design effect.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-b.png" }, { "alt": "Figure A1, 'Power of Placebo Test I to Detect Violations of Assumptions 1-3', with four shaded panels that show power against the total number of admitters, from 100 to 1000, and a proportion from 0 to 1. A red box marks the title of the lower-left panel, 'Violated assumption: no design effects', whose horizontal axis is 'Proportion of Design Affected'. In that panel the red band for power near 0.8 runs from a proportion of about 0.6 at 100 admitters to about 0.2 at 800 to 1000 admitters, much as in the two upper panels for false confessions and for liars.", "caption": "Online supplement, Figure A1, p. 3. The marked panel is the one for design effects. Its red band, which marks power near 0.80, lies near a proportion of 0.2 at 800 admitters.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-c.png" }, { "alt": "Bar chart. The horizontal axis is the share of subjects designated design-affected, at 0, 20 and 100 percent. The vertical axis is power, from 0 to 1. Blue bars, for the rule in the replication code, have heights 0.050, 0.780 and 1.000. Orange bars, for the rule described in Appendix C, have heights 0.050, 0.051 and 0.067. A horizontal line at 0.80 marks the power stated in the supplement for 20 percent of 800, and a line at 0.05 marks the level of the test.", "caption": "Power of Placebo Test I at the 5% level with 800 admitters, by the share of subjects designated design-affected, under the coded rule (blue) and the described rule (orange). Each bar is the share of rejections in 20,000 simulated experiments, with Monte Carlo standard error at most 0.003. The line at 0.80 is the power that the supplement states for 20% of 800.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E004-power.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T1", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T1" }, { "id": "P019-T2", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T2" }, { "id": "P019-T16", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T16" } ], "headline": "The simulated design effect is not the “ceiling effect” that Appendix C describes: the code adds the sensitive item after the capped increment, and against the rule as described Placebo Test I has power of about 0.05, not 0.80, with 800 admitters and 20% design-affected.", "id": "P019-E004", "location": "Online supplement, Appendix C, second paragraph and footnote 1, p. 2; the sentence on 80% power and Figure A1, lower-left panel, p. 3 (supplement PDF pp. 2–3). Replication files: ACCG_list_source.R, lines 82–86 and 103–106. The main text points to these results in Section 4.1, pp. 50–51 (PDF pp. 8–9)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Moderate", "subject": "paper", "target_ids": [ "P019-T1", "P019-T16", "P019-T2" ], "text": { "as_printed": "Supplement p.~2: ``Control list responses were drawn from a binomial distribution with a success probability of 0.4 and four trials. For those units who do meet the assumptions of No False Confessions, No Liars, or No Design Effects, treatment list responses were set equal to the control list response plus one. The treatment list responses for Liars and False Confessors were set equal to their control list responses. The treatment list response for Design Affected subjects was generated as a ``ceiling effect'': \\hl{the control list plus one, except for those with a ``4'' on the control list; those units' treatment list response remained equal to 4}.$^{1}$'' Footnote 1, attached to this sentence: ``This is one of many possible design effects; the power of our test to detect any particular design effect depends on the manner in which it changes subjects' treatment list responses.''\n\nSupplement p.~3: ``When approximately 20\\% of a sample of 800 confessors falsely confess, lie on the treatment list, \\hl{or change their responses to the non-sensitive items on the treatment list}, the placebo test achieves 80\\% power.'' The lower-left panel of Figure A1 has the title ``Violated assumption: no design effects''. Its horizontal axis is the proportion of ``Design Affected'', and its vertical axis is the ``Total Number of Admitters''.\n\nIn the file \\texttt{ACCG\\_list\\_source.R} of the replication files, lines 83--85 are the body of the function \\texttt{design\\_affector}, and lines 103--106 are in the function \\texttt{list\\_simulator}:\n\\begin{itemize}\n\\item \\texttt{x\\_star <- x}\n\\item \\texttt{x\\_star[x!=4] <- x[x!=4] + 1}\n\\item \\texttt{return(x\\_star)}\n\\item \\texttt{V0 <- rbinom(N, size=4, prob=baseline\\_binomial\\_prob)}\n\\item \\texttt{V1 <- V0+ Ystar*(1-listliar)}\n\\item \\texttt{V1[designaffected==1] <- design\\_affector(V0[designaffected==1]) +}\n\\item \\texttt{\\hl{Ystar[designaffected==1]*(1-listliar[designaffected==1])}}\n\\end{itemize}", "corrected": "Supplement p.~2: ``The treatment list response for Design Affected subjects was generated \\hl{by raising the response to the non-sensitive items by one, except for those with a ``4'' on the control list, and then adding the sensitive item. For subjects who engage in the behavior and do not lie on the list, the treatment list response is therefore the control list response plus two, or 5 for those with a ``4'' on the control list.}''\n\nSupplement p.~3: ``When approximately 20\\% of a sample of 800 confessors falsely confess, lie on the treatment list, or \\hl{report one more non-sensitive item on the treatment list unless they already report all four, as described above}, the placebo test achieves \\hl{about} 80\\% power.''\n\nIf the ceiling rule of the printed text is kept instead, the code should set the treatment-list response of a design-affected admitter to $\\min(W+1,4)$ with no further addition, the lower-left panel should be computed again, and the text should say that against this design effect the test has power of about 0.05 with 800 admitters and 20\\% design-affected.", "evidence": "The evidence is an exact comparison of the two rules and a simulation of both with the authors' test function. Running the function \\texttt{design\\_affector} unchanged and adding the sensitive item, I obtain the following responses.\n\\begin{audittable}{rrrr}{Treatment-list response of an admitter in the simulation for the lower-left panel, by control-list response $W$.}\n$W$ & Not design-affected & Design-affected, described rule & Design-affected, coded rule \\\\\n\\hline\n0 & 1 & 1 & 2 \\\\\n1 & 2 & 2 & 3 \\\\\n2 & 3 & 3 & 4 \\\\\n3 & 4 & 4 & 5 \\\\\n4 & 5 & 4 & 5 \\\\\n\\end{audittable}\nThe control-list response $W$ is binomial with four trials and success probability $2/5$, and the values $0,\\dots,4$ have probabilities $81/625$, $216/625$, $216/625$, $96/625$ and $16/625$. Let $A$ indicate a design-affected subject, and let $\\rho$ be the share of such subjects, who are drawn at random independently of $W$ and of the treatment. The treatment-list response of an admitter is $W+1-A\\,\\mathbf{1}\\{W=4\\}$ under the described rule and $W+1+A\\,\\mathbf{1}\\{W<4\\}$ under the coded rule. Taking expectations, I obtain\n\\[\n\\beta_{\\text{described}}=1-\\frac{16}{625}\\,\\rho,\\qquad \\beta_{\\text{coded}}=1+\\frac{609}{625}\\,\\rho.\n\\]\nAt $\\rho=1/5$ these values are $3109/3125=0.99488$ and $3734/3125=1.19488$.\n\nThe simulation follows the script for the panel: the stated number of admitters, 200 withholders and 200 respondents who do not engage in the behavior, with the specified share of all subjects designated design-affected. Both rules are applied to the same draws, the test is the authors' \\texttt{placeboI\\_test}, and the null is rejected when $p<0.05$.\n\\begin{audittable}{rrrr}{Power of Placebo Test I at the 5\\% level, from 20,000 simulated experiments in each row, with Monte Carlo standard errors in parentheses.}\nAdmitters & Share design-affected & Described rule & Coded rule \\\\\n\\hline\n800 & 0 & 0.05015 (0.00154) & 0.05015 (0.00154) \\\\\n800 & 0.20 & 0.05070 (0.00155) & 0.77965 (0.00293) \\\\\n800 & 1 & 0.06710 (0.00177) & 1.00000 \\\\\n1000 & 1 & 0.06950 (0.00180) & 1.00000 \\\\\n\\end{audittable}\nIn the last two rows the coded rule led to rejection in all 20,000 experiments. In a second simulation, written independently and run with another seed, the power is 0.05145 and 0.77780 at 800 admitters and a share of 0.20. The full grid of Figure A1 was not computed again, and the values of its cells were not recovered. The simulation does not show that the power under the described rule stays near 0.05 in arbitrarily large samples, since $\\beta_{\\text{described}}$ differs from 1 whenever $\\rho>0$.", "problem": "Appendix C says that the treatment-list response of a design-affected subject is the control-list response plus one, capped at four. I call this the described rule. I write $W$ for the control-list response, and I call a respondent who answers ``Yes'' to the direct question an admitter, as Figure A1 does. In the simulation for the lower-left panel of Figure A1, no admitter confesses falsely or lies on the list. For a design-affected admitter, the response under the described rule is $\\min(W+1,4)$, that is, $1,2,3,4,4$ for $W=0,\\dots,4$. The replication code does something else. The function \\texttt{design\\_affector} raises the count of non-sensitive items by one unless it is already four, and the simulator then adds the sensitive item. I call this the coded rule. Under the coded rule the response is therefore $\\min(W+1,4)+1$, that is, $2,3,4,5,5$. The two rules differ at every value of $W$.\n\nThe two rules lead to different power. Placebo Test I tests whether the difference $\\beta$ between the mean responses of the two arms among admitters equals 1. Under the described rule a design-affected admitter differs from an unaffected one only when $W=4$, an event of probability $0.4^4=0.0256$. Then $\\beta$ falls below 1 by $0.0256$ times the affected share. Under the coded rule $\\beta$ rises above 1 by $0.9744$ times that share. The lower-left panel of Figure A1 and the last clause of the sentence on 80\\% power report high power. The coded rule has it and the described rule does not. With 800 admitters and 20\\% design-affected, the power is about 0.78 under the coded rule and about 0.05, the level of the test, under the described rule. Footnote 1 says that power depends on the form of the design effect. The statement of 80\\% power is then read as a statement about the rule just described. The evidence cannot decide which of the two the authors intended, the description or the code.", "what_survives": "The defect is confined to three places: the description of the design effect in Appendix C, the reading of the lower-left panel of Figure A1 as that design effect, and the last clause of the sentence on 80\\% power. The example in the main text, with about 80 percent power when 20 percent of 800 are false confessors, concerns false confessions and is not affected. The test formula of Proposition 3, the identification and efficiency results, the other panels of Figure A1, Placebo Test II, and the estimates and placebo p-values of the application do not depend on the simulated design effect. The high-power conclusion does not hold for the printed ceiling rule. Either the statement of 80\\% power is attached to the positive design effect that was simulated, or the ceiling rule is kept and the panel shows power near the level of the test at the point that the text cites." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E004", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "The two upper panels of Figure A2, 'Power of Placebo Test II to Detect Violations of Treatment Independence'. A red box marks the title 'Power of Placebo Test II, N= 100' of the left panel. The right panel is titled 'N= 200'. Each panel shades power from white, for 0, to black, for 1, against Pr(Y=1|Z=1) on the horizontal axis and Pr(Y=1|Z=0) on the vertical axis, with a pale band along the diagonal and a red band for power near 0.8 on either side of it.", "caption": "Online supplement, Figure A2, upper panels, p. 4. The marked panel, for $N=100$, is computed without the opposite rejection tail, and the panel beside it, for $N=200$, includes that tail. The omitted tail is largest on the nondegenerate diagonal, where the first panel plots 0.025 and the second 0.05.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E006.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T15", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T15" } ], "headline": "The panel of Figure A2 for $N=100$ leaves out the rejection tail opposite to the true difference, which the other three panels count; it understates the two-sided power by at most 0.025.", "id": "P019-E006", "location": "Online supplement, Appendix D, p. 3, and Figure A2, first panel, p. 4 (supplement PDF pp. 3–4). Replication files: ACCG_list_placebo_II_power.R, line 31", "pdf_page": 3, "printed_page": "3", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T15" ], "text": { "as_printed": "Supplement p.~3: ``The power of Placebo Test II to detect violations of Treatment Independence is equivalent to the power of test of a difference in proportions. Following the implementation in R (see accompanying text: Dalgaard 2008, p. 159), the power of the test for $N = 100$, $200$, $500$, and $1000$ is shown in Figure A2 below.'' Figure A2, first panel: ``Power of Placebo Test II, $\\hl{N=100}$''.\n\nReplication files, \\texttt{ACCG\\_list\\_placebo\\_II\\_power.R}. Line 31, for $N=100$, and line 62, for $N=200$, compute the power with the following two calls:\n\\begin{itemize}\n\\item \\texttt{\\hl{power.prop.test(n=50, p1=pr.yes.treat, p2=pr.yes.control)}}\n\\item \\texttt{power.prop.test(n=100, p1=pr.yes.treat, p2=pr.yes.control, strict=TRUE)}\n\\end{itemize}\nLines 93 and 124, for $N=500$ and $N=1000$, have the call of line 62 with \\texttt{n=250} and \\texttt{n=500}.", "corrected": "Line 31 should compute the power with the following call, and the first panel should be drawn from it.\n\\begin{itemize}\n\\item \\texttt{power.prop.test(n=50, p1=pr.yes.treat, p2=pr.yes.control\\hl{, strict=TRUE})}\n\\end{itemize}\nThe description of the figure should read: ``\\hl{Figure A2 shows normal-approximation planning power for a two-sided test of a difference in proportions at the 5\\% level, with equal treatment and control group sizes and both rejection tails included in every panel.}''", "evidence": "The evidence is a derivation of the omitted term, its exact value at one plotted point, and an evaluation of the original calls at every plotted point. Let each group have size $n$, let $p_1$ and $p_2$ be the two proportions, and set $d=|p_1-p_2|$, $S=p_1(1-p_1)+p_2(1-p_2)$, $V=(p_1+p_2)(1-(p_1+p_2)/2)$ and $c=\\Phi^{-1}(0.975)$. Suppose that $S>0$. In the normal approximation of the function, the probability of rejection in the direction of the true difference is $A$ and the probability of rejection in the opposite direction is $B$, where\n\\[\nA=\\Phi\\left(\\frac{\\sqrt n\\,d-c\\sqrt V}{\\sqrt S}\\right),\\qquad B=\\Phi\\left(\\frac{-\\sqrt n\\,d-c\\sqrt V}{\\sqrt S}\\right).\n\\]\nLine 31 plots $A$, and the other three panels plot $A+B$. Since $V-S=(p_1-p_2)^2/2\\ge0$, I have $(\\sqrt n\\,d+c\\sqrt V)/\\sqrt S\\ge c$, which implies that\n\\[\n00$, with $S$ defined below, and it is at most 0.025, the value that it takes when the two proportions are equal and lie strictly between 0 and 1. On that diagonal the first panel plots 0.025 where the other panels plot 0.05.", "what_survives": "The other three panels, the red band near 0.80 on the plotted grid, and the definition of the test in Proposition 4 are unaffected. No empirical p-value uses this function: the replication code computes the p-values of Placebo Test II directly as $2\\Phi(-|\\hat\\delta|/\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\delta]})$ in a separate function. The correction stays within the normal approximation and does not make the figure an exact calculation of power in finite samples." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E006", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the last paragraph of p. 50. Red boxes mark the sentence 'We also explore the power of Placebo Test I using a series of Monte Carlo simulations.' The paragraph continues with the factors that are varied, the example of about 30 percent power, and the beginning of the example with 800 subjects.", "caption": "Section 4.1, last paragraph, p. 50 (PDF p. 8). The marked sentence introduces the Monte Carlo power study of Placebo Test I. The faults are in the script deposited for that study and not on this page.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E005.png" } ], "formal_release": null, "formal_statements": [], "headline": "The deposited script for Figure A1, the power study of Placebo Test I, cannot run as written, and the script for Tables A2–A8 calls a package that it does not load; no published number is shown to be wrong.", "id": "P019-E005", "location": "Replication files (Harvard Dataverse, doi:10.7910/DVN/29984, version 1.0): the script of source functions, the script for the power study of Placebo Test I and the script for the replication experiment, at the lines quoted below. In the paper: Section 4.1, last paragraph, p. 50 (PDF p. 8), and Figure A1 of the online supplement, p. 3", "pdf_page": null, "printed_page": null, "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "The file \\texttt{readme.txt} says: ``This replication archive contains the data and programs necessary to replicate the figures and tables presented in the main text and the appendix.'' Its entry for \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R} has the three lines ``This file conducts the power analysis of Placebo Test I.'', ``Inputs: ACCG\\_list\\_source'' and ``Outputs: Figure A1''.\n\nIn \\texttt{ACCG\\_list\\_source.R}, line 68 is the last line of the function \\texttt{placeboI\\_test}, and line 112 is in the function \\texttt{list\\_simulator}:\n\\begin{itemize}\n\\item \\texttt{return(c(est=est, se=se, p\\_value=p\\_value, n=n))}\n\\item \\texttt{reject\\_null <- as.numeric(placeboI\\_test(V=V, Z=Z, Y=Y)\\hl{\\$p\\_value}<0.05)}\n\\end{itemize}\nLine 88 begins the definition of \\texttt{list\\_simulator}. The first four arguments of this function are named \\texttt{N\\_trueadmitter}, \\texttt{N\\_falseconfessor}, \\texttt{N\\_withholder} and \\texttt{N\\_innocent}. The other three are named \\texttt{pr\\_listliar}, \\texttt{pr\\_designaffected} and \\texttt{baseline\\_binomial\\_prob}.\n\nIn \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R}, line 41 calls \\hl{\\texttt{listsimulator}} with the named arguments \\hl{\\texttt{N.trueadmitter}}, \\hl{\\texttt{N.falseconfessor}}, \\hl{\\texttt{N.withholder}}, \\hl{\\texttt{N.innocent}} and \\texttt{baseline\\_binomial\\_prob}. Lines 83, 124 and 166 have the same form. Line 83 adds the argument \\hl{\\texttt{pr.listliar}}, and line 124 adds the argument \\hl{\\texttt{pr.designaffected}}.\n\nIn \\texttt{ACCG\\_list\\_replication\\_experiment.R}, line 6, \\texttt{library(lmtest)}, is the only line that loads a package, and the function \\texttt{xtable} is called at lines 45, 58, 73, 88, 100, 113 and 142.", "corrected": "In \\texttt{ACCG\\_list\\_placebo\\_I\\_power.R}, lines 41, 83, 124 and 166 should call \\hl{\\texttt{list\\_simulator}} with the argument names \\hl{\\texttt{N\\_trueadmitter}}, \\hl{\\texttt{N\\_falseconfessor}}, \\hl{\\texttt{N\\_withholder}}, \\hl{\\texttt{N\\_innocent}}, and, where used, \\hl{\\texttt{pr\\_listliar}} and \\hl{\\texttt{pr\\_designaffected}}. Line 112 of \\texttt{ACCG\\_list\\_source.R} should read as follows.\n\\begin{itemize}\n\\item \\texttt{reject\\_null <- as.numeric(placeboI\\_test(V=V, Z=Z, Y=Y)\\hl{[[\"p\\_value\"]]}<0.05)}\n\\end{itemize}\nThe function \\texttt{placeboI\\_test} should keep its numeric vector as return value, since the code for the tables binds these vectors as rows. The script \\texttt{ACCG\\_list\\_replication\\_experiment.R} should load the package with \\hl{\\texttt{library(xtable)}}. The user must also set the working directory and arrange the folders for programs and data, as the comments in the scripts ask. To make new parallel simulations repeatable, the script should fix the seed and initialize the random streams of the workers, for example with \\texttt{clusterSetRNGStream}; a call to \\texttt{set.seed} alone does not reach the workers. A new seed cannot recover the published draws.", "evidence": "The evidence is a set of executable checks on the unchanged files, each in a new R session. The four calls to the simulator report \\texttt{could not find function \"listsimulator\"}. With that name pointed at \\texttt{list\\_simulator}, they report unused arguments. A direct call of \\texttt{list\\_simulator} reports \\texttt{\\$ operator is invalid for atomic vectors}. A small example makes the last failure exact. For $V=(0,2,1,3)$, $Z=(0,0,1,1)$ and $Y=(1,1,1,1)$, the function \\texttt{placeboI\\_test} returns a numeric vector with estimate $1$, standard error $\\sqrt2$, p-value $1$ and $n=4$, and \\texttt{\\$p\\_value} fails on it. With the directories set, the script for Tables A2--A8 stops at line 45 with \\texttt{could not find function \"xtable\"}. With the directories set and \\texttt{xtable} made available, and with the estimator code unchanged, I find that all 360 numeric entries of Tables 3--8 and of Tables A1--A8 of the supplement agree with the publication at the printed precision.", "problem": "The readme says that the archive contains the programs needed to replicate the figures and tables. The two power scripts and the two analysis scripts each begin with a call that sets the working directory, and a comment asks the user to set it. The readme describes a layout with folders for programs and data. This is ordinary setup. After the setup, the script for Figure A1 still cannot run. Its four calls to the simulator use the name \\texttt{listsimulator}, which is not defined, and argument names with dots. The function supplied is \\texttt{list\\_simulator}, and its argument names have underscores. The supplied function itself stops on every call that reaches line 112, since \\texttt{placeboI\\_test} returns a named numeric vector and \\texttt{\\$p\\_value} cannot be extracted from such a vector. The script for Tables A2--A8 stops at its first call to \\texttt{xtable}, a package that it never loads. The script for Figure A1 also sets no seed, and the Monte Carlo draws of the published figure cannot be repeated.", "what_survives": "These faults do not show that any theorem, estimator formula, estimate or power value is wrong. The functions that produce the empirical tables do not call the simulator, and the 360 table entries compared here agree. Two points are not established: which version of the code drew the published Figure A1, and whether the full grid of that figure is recovered once the script runs. The full grid was not computed again." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E005", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the CNN row of Table A1, Survey Questions and Response Options. Red boxes mark the words 'read or watch' in the list question, 'select how many you read or watch in the course of an ordinary month', and the words 'do you watch CNN?' in the direct question. The list items are The New York Times, CNN in bold, The Huffington Post, Fox News and Politico; the response options to the direct question are Yes and No.", "caption": "Table A1, p. 61 (PDF p. 19), row “CNN (L5 & D5)”. The list question asks about reading or watching; the direct question asks about watching only.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E007.png" }, { "alt": "Crop of a paragraph of Section 5.2. A red box, which runs over two lines, marks the words 'providing confidence that the list experiments and the direct questions are measuring the same quantities' in the sentence that introduces Figure 1.", "caption": "Section 5.2, p. 54 (PDF p. 12). The marked remark takes agreement of the estimates as a sign that the two kinds of question measure the same quantities.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E007-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T13", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T13" }, { "id": "P019-T14", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T14" } ], "headline": "Because the list question for CNN asks which organizations respondents “read or watch” and the direct question asks only whether they “watch CNN”, a direct “No” need not be withholding, and the list-based estimates for CNN may measure the broader behaviour; the CNN comparison needs this qualification.", "id": "P019-E007", "location": "Appendix A.1, Table A1, p. 61 (PDF p. 19), row “CNN (L5 & D5)”; Section 5.2, p. 54 (PDF p. 12)", "pdf_page": 19, "printed_page": "61", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T13", "P019-T14" ], "text": { "as_printed": "Table A1, row ``CNN (L5 \\& D5)''. List question: ``Here is a list of four (five) news organizations. Please read all four (five) and select how many you \\hl{read or watch} in the course of an ordinary month. We do not want to know which ones, just how many.'' Items: \\textit{The New York Times}; \\textbf{CNN}; The Huffington Post; Fox News; Politico. Direct question: ``In the course of an ordinary month, do you \\hl{watch} CNN?'' Response options: Yes; No.\n\nSection 5.2, p.~54: ``Figure 1 presents these results graphically: the estimates generally agree (providing confidence that the list experiments and the direct questions are measuring the same quantities), and the 95 percent confidence intervals around the combined estimate are always tighter than those around the standard estimate.''", "corrected": "Add to the discussion of the CNN results in Section 5.2: ``For CNN, the list question asks which organizations respondents \\hl{read or watch}, whereas the direct question asks only whether they \\hl{watch} CNN. A person who reads CNN but does not watch it can therefore answer both questions truthfully, be counted by the list and answer `No' to the direct question; that discrepancy need not be withholding about watching. Under the ordinary reading of the list question, \\hl{if the identifying assumptions hold for reading or watching CNN}, the standard list and combined estimates target that broader behaviour.''\n\nThe questions stay as printed in Table A1, since they are the questions that were asked. Future surveys should use the same wording for the sensitive item in the list question and in the direct question.", "evidence": "The evidence is a check of the page and an exact example. Suppose that one-quarter of the population watches CNN and says so, one-quarter reads CNN without watching it, and one-half does neither. Everyone answers both questions truthfully, the baseline count is 1 or 2 with equal probability, and assignment is by a fair coin independent of everything else. Then the mean direct answer is $1/4$, which is the share that watches CNN, while the standard list difference and the combined expression (1) both equal $1/2$, the share that reads or watches. With $X_i$ taken as reading or watching, the numerical conditions of Assumptions 1--4 hold, and so do the null hypotheses of both placebo tests ($\\beta=1$ and $\\delta=0$). If those who only read CNN are replaced by watchers who answer ``No'', then the joint distribution of assignment, direct answer and list count is unchanged, and the responses cannot distinguish the two cases. If those who only read CNN are 1 percent of the population, then the mean direct answer is $1/4$ and the two list-based quantities are $13/50$. Near agreement of the estimates does not show that the questions measure the same behaviour. The example shows what the wording allows. It does not show that such respondents were in the sample, and no reported CNN estimate is shown to be wrong.", "problem": "The paper's framework has one behaviour $X_i$ for each topic: the list counts it for treated respondents, and the direct question asks about it. For CNN the two questions do not ask about the same behaviour. Under the ordinary reading of ``read or watch'', a respondent counts CNN in the list if she reads it or watches it, and she answers ``Yes'' to the direct question only if she watches it. A respondent who reads CNN and does not watch it answers both questions truthfully when she counts CNN in the list and answers ``No''. Therefore, for CNN, a gap between the list-based estimates and the direct estimate need not reflect withholding, and the standard list and combined estimates may measure reading or watching CNN, which is a broader behaviour than the one in the direct question. The remark in Section 5.2 is tentative, but for CNN the wording of the questions does not support it. The paper does not show how respondents understood the list question, and some may have taken ``read or watch'' to mean reading newspapers and watching broadcasters.", "what_survives": "The identification and efficiency results, which assume a single behaviour $X_i$, are unaffected, and so are the other four topics. The CNN entries of Tables 3 and 6 and of Figures 1 and 2, the placebo tests and the replicated effect of the treatment list on direct answers about CNN stand as statements about the responses. Their interpretation for CNN changes in the tables and figures, in the replication, and in supplement Appendix E, which describes the standard list estimate as an estimate of ``those watching CNN''. The paper already reports that CNN fails Placebo Test I in Study A ($p=0.034$) and Placebo Test II in Study B ($p=0.003$), and it calls the combined estimate for CNN in Study B unreliable." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E007", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of the proof of Lemma 1. A red box marks 'W_i + mu - W_i' in equation (2), which reads E[V_i | Z_i = 1] - E[V_i | Z_i = 0] = W_i + mu - W_i = mu. A second red box marks 'Assumption 2.' at the end of the sentence 'The result follows since E[X_i | Y_i = 1] = 1 by Assumption 2.'", "caption": "Proof of Lemma 1, p. 48 (PDF p. 6). The first mark is on the middle expression of equation (2), where $W_i$ stands for $\\mathrm{E}[W_i]$. The second is on the citation of Assumption 2 for a consequence of Monotonicity.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008.png" }, { "alt": "Crop of Corollary 1: 'If Assumptions 1-4 hold, then confidence intervals constructed as mu-hat plus or minus z times the square root of the estimated variance will have mu 100(1 - alpha) percent coverage for mu for large n.' A red box marks the first mu, which stands directly before '100(1 - alpha)', and the percent sign follows.", "caption": "Corollary 1, p. 49 (PDF p. 7). The marked $\\mu$ before the coverage percentage does not belong to the sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-b.png" }, { "alt": "Crop of the start of the proof of Lemma 2, with the sentence 'Suppose that exactly k direct responses are zero', display (4), whose last line conditions on 'Y_j = 0 for j = 1,...,k', and the sentence 'where we have reordered the indices so that Y_1 = ... = Y_k = 1.' A red box marks 'Y_1 = ... = Y_k = 1.'", "caption": "Appendix B, proof of Lemma 2, p. 62 (PDF p. 20). Display (4) conditions on $Y_j=0$ for $j=1,\\dots,k$, and the marked phrase gives the same responses the value 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-c.png" }, { "alt": "Crop of two lines of the proof of Proposition 2, 'By Assumptions 1 and 2 (Monotonicity, No Liars, and No Design Effects), E[V_i | Z_i = 1, Y_i = 1] = E[V_i | Z_i = 0, Y_i = 1] + 1. Then,', followed by the start of a display. A red box marks 'Assumptions 1 and 2'.", "caption": "Appendix B, proof of Proposition 2, p. 64 (PDF p. 22). The marked citation omits Assumption 3.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E008-d.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T11", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T11" }, { "id": "P019-T19", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T19" } ], "headline": "Five local slips in the derivations: equation (2) should contain $\\mathrm{E}[W_i]$ in place of $W_i$, Monotonicity is Assumption 1 and not Assumption 2, Corollary 1 has a stray $\\mu$, the reordered responses in the proof of Lemma 2 are zeros, and one step in the proof of Proposition 2 also uses Assumption 3.", "id": "P019-E008", "location": "Proof of Lemma 1, equation (2) and last sentence, p. 48 (PDF p. 6); Corollary 1, p. 49 (PDF p. 7); Appendix B, proof of Lemma 2, sentence after display (4), p. 62 (PDF p. 20); Appendix B, proof of Proposition 2, p. 64 (PDF p. 22)", "pdf_page": 6, "printed_page": "48", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T11", "P019-T19" ], "text": { "as_printed": "\\begin{enumerate}\n\\item p.~48, proof of Lemma 1: ``By Assumptions 2 and 3,''\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]-\\mathrm{E}[V_i\\mid Z_i=0]=\\hl{W_i}+\\mu-\\hl{W_i}=\\mu.\\qquad(2)\n\\]\n\\item p.~48, end of the same proof: ``The result follows since $\\mathrm{E}[X_i\\mid Y_i=1]=1$ by \\hl{Assumption 2}.''\n\\item p.~49: ``Corollary 1. If Assumptions 1--4 hold, then confidence intervals constructed as $\\hat\\mu\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\mu]}$ will have $\\hl{\\mu}$ $100(1-\\alpha)\\%$ coverage for $\\mu$ for large $n$.''\n\\item p.~62, proof of Lemma 2, after display (4): ``where we have reordered the indices so that $Y_1=\\cdots=Y_k=\\hl{1}$.''\n\\item p.~64, proof of Proposition 2: ``By \\hl{Assumptions 1 and 2} (Monotonicity, No Liars, and No Design Effects), $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.''\n\\end{enumerate}", "corrected": "\\begin{enumerate}\n\\item p.~48: ``By Assumptions 2 and 3,''\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]-\\mathrm{E}[V_i\\mid Z_i=0]=\\hl{\\mathrm{E}[W_i]}+\\mu-\\hl{\\mathrm{E}[W_i]}=\\mu.\\qquad(2)\n\\]\n\\item p.~48: ``The result follows since $\\mathrm{E}[X_i\\mid Y_i=1]=1$ by \\hl{Assumption 1}.''\n\\item p.~49: ``Corollary 1. If Assumptions 1--4 hold, then confidence intervals constructed as $\\hat\\mu\\pm z_{1-\\alpha/2}\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\mu]}$ will have $100(1-\\alpha)\\%$ coverage for $\\mu$ for large $n$.'' The $\\mu$ before the percentage is deleted.\n\\item p.~62: ``where we have reordered the indices so that $Y_1=\\cdots=Y_k=\\hl{0}$.''\n\\item p.~64: ``By \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, \\hl{and Treatment Independence}), $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.''\n\\end{enumerate}", "evidence": "The evidence is a check of the pages against the definitions of the assumptions on pp.~46--47, and a derivation of the identities in (a) and (e). For (a), recall that Assumption 2 states that $V_i=W_i+X_iZ_i$. By Assumption 3, the pair $(W_i,X_i)$ is independent of $Z_i$, and I have\n\\[\n\\mathrm{E}[V_i\\mid Z_i=1]=\\mathrm{E}[W_i]+\\mathrm{E}[X_i]=\\mathrm{E}[W_i]+\\mu,\\qquad \\mathrm{E}[V_i\\mid Z_i=0]=\\mathrm{E}[W_i].\n\\]\nFor (e), suppose that $Y_i=1$. Then Assumption 1 implies that $X_i=1$, and by Assumption 2 I have $V_i=W_i+Z_i$. By Assumption 3, the conditional distribution of $W_i$ given $Y_i=1$ is the same in the two arms, so that $\\mathrm{E}[W_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[W_i\\mid Z_i=0,Y_i=1]$. I conclude that $\\mathrm{E}[V_i\\mid Z_i=1,Y_i=1]=\\mathrm{E}[V_i\\mid Z_i=0,Y_i=1]+1$.", "problem": "(a) The middle expression of equation (2) is a difference of two expectations. It should contain the mean $\\mathrm{E}[W_i]$ of the baseline response and not the random variable $W_i$. The two terms cancel in either form, and the value $\\mu$ is unaffected. (b) The equality $\\mathrm{E}[X_i\\mid Y_i=1]=1$ says that no respondent confesses falsely, which is Monotonicity (Assumption 1). Assumption 2 is No Liars and No Design Effects, and the sentence after the proof calls Assumption 1 Monotonicity. (c) The $\\mu$ before the coverage percentage does not belong to the sentence. (d) Display (4) conditions on $Y_j=0$ for $j=1,\\dots,k$, and the paragraph begins ``Suppose that exactly $k$ direct responses are zero''. The reordered responses are therefore the zeros. (e) The step equates the baseline means of the two arms among respondents with $Y_i=1$, which uses Treatment Independence (Assumption 3) as well as Assumptions 1 and 2. All three are hypotheses of Proposition 2.", "what_survives": "Nothing depends on these slips. The identity (1) of Lemma 1, the coverage statement of Corollary 1 and the inequality of Proposition 2 are unchanged, and so are the estimator and the reported analyses. Other steps of the proofs in Appendix B are the subject of \\fid{P019-E003}." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E008", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "page_check", "exhibits": [ { "alt": "Crop of Table 5, Study A (Directs First): Placebo Test II, with columns for the estimate, SE, p-value and n, and five rows from Nuclear power to CNN. A red box marks the header of the estimate column, which is printed as beta-hat. The estimates below it are 0.066, -0.000, 0.016, -0.030 and -0.056, each with n equal to 500.", "caption": "Table 5, p. 56 (PDF p. 14). The marked header names the statistic of Placebo Test I; the entries below it are the estimates $\\hat\\delta$ of Placebo Test II.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009.png" }, { "alt": "Crop of supplement Tables A3 and A4. Both captions read 'Study A (Direct First): Placebo Test I'. Red boxes mark the words 'Placebo Test I' in the caption of Table A4 and the beta-hat header of its estimate column. Table A3 has estimates near 1 and n between 31 and 327; Table A4 has estimates near 0 and n equal to 506 in every row.", "caption": "Supplement, p. 6. Table A4 repeats the caption “Placebo Test I” of Table A3 above it, and its estimate column is headed $\\hat\\beta$; its entries, all with $n=506$, are the differences of Placebo Test II.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-c.png" }, { "alt": "Crop of a paragraph of Section 5.3. A red box marks the words 'direction question' in the sentence 'In Study B, subjects were exposed to either a treatment or a control list before answering the direction question.' The next sentence refers to answers to the direct questions.", "caption": "Section 5.3, p. 57 (PDF p. 15). The marked words should read “direct question”.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-b.png" }, { "alt": "Crop of the first paragraph of supplement Appendix D. A red box marks the citation 'Dalgaard 2008, p. 159' in the sentence 'Following the implementation in R (see accompanying text: Dalgaard 2008, p. 159), the power of the test for N = 100, 200, 500, and 1000 is shown in Figure A2 below.'", "caption": "Supplement Appendix D, p. 3. The marked citation has no entry in the reference list of the article, and the supplement has no reference list.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E009-d.png" } ], "formal_release": null, "formal_statements": [], "headline": "The four tables of Placebo Test II head their estimate column $\\hat\\beta$ where $\\hat\\delta$ is meant, supplement Table A4 is captioned Placebo Test I where it reports Placebo Test II, “direction question” should read “direct question”, and the citation of Dalgaard (2008) has no reference entry; every number in these tables agrees with a recalculation from the replication files.", "id": "P019-E009", "location": "Table 5, p. 56 (PDF p. 14), and Table 8, p. 58 (PDF p. 16), estimate column; Section 5.3, p. 57 (PDF p. 15); supplement Tables A4 and A7, pp. 6–7, and Appendix D, p. 3; analysis scripts in the replication files", "pdf_page": 14, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [], "text": { "as_printed": "\\begin{itemize}\n\\item Table 5, p.~56: ``Table 5. Study A (Directs First): Placebo Test II'', with the estimate column headed $\\hl{\\hat\\beta}$. Table 8, p.~58, and supplement Tables A4 and A7 carry the same header.\n\\item Supplement, p.~6: ``Table A4: Study A (Direct First): \\hl{Placebo Test I}''. The table above it is ``Table A3: Study A (Direct First): Placebo Test I''.\n\\item Section 5.3, p.~57: ``In Study B, subjects were exposed to either a treatment or a control list before answering the \\hl{direction} question.''\n\\item Supplement Appendix D, p.~3: ``Following the implementation in R (see accompanying text: \\hl{Dalgaard 2008, p. 159}), the power of the test for $N = 100$, $200$, $500$, and $1000$ is shown in Figure A2 below.''\n\\item Replication files: the scripts for the main and the replication experiment label the p-value column of Placebo Test I ``Pr($\\beta\\hl{=0}$)'', typed as a formula in LaTeX notation, in four places.\n\\end{itemize}", "corrected": "\\begin{itemize}\n\\item Tables 5, 8, A4 and A7: head the estimate column $\\hl{\\hat\\delta}$.\n\\item ``Table A4: Study A (Direct First): \\hl{Placebo Test II}''.\n\\item ``In Study B, subjects were exposed to either a treatment or a control list before answering the \\hl{direct} question.''\n\\item Add a complete reference entry for Dalgaard (2008), after checking which work is meant and the page cited.\n\\item In the four places in the scripts: \\texttt{colnames(placebo\\_I\\_table) <- c(\"Estimate\", \"SE\", \"p-value (H0: beta = 1)\", \"N\")}.\n\\end{itemize}", "evidence": "The evidence is a check of the pages and a recalculation from the replication files. All 80 entries of the four tables (estimates, standard errors, p-values and sample sizes) agree with the calculation of Placebo Test II at the printed precision, and each of the 20 estimates differs from the corresponding $\\hat\\beta$. For nuclear power in Study A, for example, $\\hat\\delta=276/4165=0.066$, the first entry of Table 5, while $\\hat\\beta=28321/26871=1.054$, the first entry of Table 4. I could not determine from the article, the supplement or the replication files which work the Dalgaard citation refers to, or check its page number.", "problem": "Sections 4.1 and 4.2 define two statistics. Placebo Test I uses $\\hat\\beta$, the difference in mean list counts between treatment and control among those who answer ``Yes'' to the direct question, with null value $\\beta=1$. Placebo Test II uses $\\hat\\delta$, the difference between treatment and control in the proportion who answer ``Yes'', with null value $\\delta=0$. Tables 5, 8, A4 and A7 report Placebo Test II and head the estimate column with the symbol of Placebo Test I. Table A4 also carries the caption of Table A3. The word ``direction'' is a slip for ``direct''. The citation of Dalgaard (2008) has no entry in the reference list on pp.~65--66, and the supplement has no reference list. In the scripts the label of the p-value column of Placebo Test I names the null $\\beta=0$, while the function computes the two-sided p-value for $\\beta=1$, as the paper states. Further, the label is written with a single backslash before \\texttt{beta}, which R reads together with the letter b as a backspace character.", "what_survives": "Every estimate, standard error, p-value and sample size in the placebo tables stands, and so do the conclusions drawn from them, including the significant results of Placebo Test II for nuclear power and CNN in Study B. The corrections change headers, one caption, one word, one reference and a label in the code." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E009", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of Figure 1, a bar chart of the direct, standard list and combined estimates of prevalence for five topics, with interval whiskers, on a vertical axis from 0.0 to 1.0. A red box marks the foot of the standard list and combined bars for Spanish-speaking, where the two whiskers run into the zero axis without the horizontal cap that ends every other whisker.", "caption": "Figure 1, p. 55 (PDF p. 13). The marked whiskers, for the standard list and combined estimates of Spanish speaking, end at the zero axis with no lower cap; the computed lower limits are $-0.121$ and $-0.102$.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E010.png" }, { "alt": "Crop of Figure 2, a bar chart of the direct, standard list and combined estimates of prevalence for five topics in Study B, with interval whiskers, on a vertical axis from 0.0 to 1.0. Two red boxes mark the feet of the standard list and combined bars for Spanish-speaking and for Muslim Teachers, where four whiskers run into the zero axis without a lower cap.", "caption": "Figure 2, p. 57 (PDF p. 15). The four marked whiskers, for the standard list and combined estimates of Spanish speaking and of Muslim teachers, end at the zero axis with no lower cap.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E010-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "Six lower confidence limits in Figures 1 and 2 are negative, and the figures cut them off at the zero axis without saying so; the intervals are not truncated in the calculation and follow from the estimates and standard errors of Tables 3 and 6.", "id": "P019-E010", "location": "Figure 1, p. 55 (PDF p. 13), and Figure 2, p. 57 (PDF p. 15): the lower ends of the interval whiskers at the zero axis", "pdf_page": 13, "printed_page": "55", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``Figure 1. Study A (Directs First): Three Estimates of Prevalence.''\n\n``Figure 2. Study B (Lists First): Three Estimates of Prevalence.''\n\nBoth figures have a vertical axis labelled ``Prevalence'' with ticks from 0.0 to 1.0. The whiskers of the standard list and combined estimates for Spanish speaking in Figure 1, and for Spanish speaking and Muslim teachers in Figure 2, \\hl{run down to the zero axis and have no lower cap}.", "corrected": "``Figure 1. Study A (Directs First): Three Estimates of Prevalence. \\hl{Lower Wald confidence limits below zero are clipped at the plotting boundary.}''\n\n``Figure 2. Study B (Lists First): Three Estimates of Prevalence. \\hl{Lower Wald confidence limits below zero are clipped at the plotting boundary.}''\n\nAlternatively, the vertical axis of both figures can be extended below zero so that the complete intervals are shown.", "intended_meaning": "The figures show each estimate with a 95 percent interval, the estimate plus or minus 1.96 standard errors, and six of these intervals have a negative lower limit. The plotting code in the replication files draws the full interval on an axis limited to the range from 0 to 1, and the plotting region cuts the whisker off at zero. The missing lower cap is the only sign of this, and the captions and the text do not mention it. Nothing printed is false, and the intervals are not truncated in the calculation. Recalculating them from the replication files, I find the limits below.\n\\begin{audittable}{llrrrr}{The six intervals whose lower limit lies below the zero axis.}\nStudy and topic & Estimator & Estimate & SE & Lower limit & Upper limit \\\\\nA, Spanish speaking & Standard list & 0.035 & 0.079 & $-0.1207$ & 0.190 \\\\\nA, Spanish speaking & Combined & 0.042 & 0.074 & $-0.1017$ & 0.187 \\\\\nB, Spanish speaking & Standard list & 0.144 & 0.083 & $-0.0192$ & 0.307 \\\\\nB, Spanish speaking & Combined & 0.149 & 0.076 & $-0.0006$ & 0.298 \\\\\nB, Muslim teachers & Standard list & 0.107 & 0.083 & $-0.0554$ & 0.268 \\\\\nB, Muslim teachers & Combined & 0.123 & 0.074 & $-0.0218$ & 0.268 \\\\\n\\end{audittable}\nSince the estimates and standard errors agree with Tables 3 and 6, the limits can be recovered from those tables to within rounding. The fourth limit is so close to zero that the rounded entries of Table 6 give a value just above zero." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E010", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "clarity_note", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "recalculation_from_source", "exhibits": [ { "alt": "Crop of a paragraph of Section 5.1. A red box, which runs over two lines, marks the words '(with the exception of the diagonal)' in the sentence 'Consistent with our randomization procedure, each cell in the table (with the exception of the diagonal) contains approximately one-quarter of the subjects.' The sentence before it says that all randomizations used Bernoulli assignment with probability 0.5.", "caption": "Section 5.1, p. 53 (PDF p. 11), first paragraph below Table 2. The marked words are the only exception that the sentence makes to its statement about the cells of Table 1.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E011.png" }, { "alt": "Crop of Table 1, Number of Subjects in Each Treatment Condition, a triangular table that crosses Study A or B and the treatment and control conditions of Lists 1 to 5. Twelve red boxes mark the zeros printed in the six two-by-two blocks that cross a factor with itself, for example 500, 0 over 0, 514 for Study A or B and 517, 0 over 0, 497 for List 1. Cells crossing distinct factors hold counts between 219 and 278.", "caption": "Table 1, p. 52 (PDF p. 10). The twelve marked zeros lie beside the diagonal, in the blocks that cross a factor with itself, and the printed exception does not cover them.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-E011-b.png" } ], "formal_release": null, "formal_statements": [], "headline": "The statement that each cell of Table 1 outside the diagonal holds about one-quarter of the subjects is meant for the cells that cross two different randomized factors; the cells that pair the two levels of one factor are zero by construction.", "id": "P019-E011", "location": "Section 5.1, p. 53 (PDF p. 11), first paragraph below Table 2; the sentence describes Table 1, p. 52 (PDF p. 10)", "pdf_page": 11, "printed_page": "53", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": null, "subject": "paper", "target_ids": [], "text": { "as_printed": "``All randomizations used Bernoulli random assignment with equal probability 0.5. Consistent with our randomization procedure, each cell in the table \\hl{(with the exception of the diagonal)} contains approximately one-quarter of the subjects.''\n\nTable 1, p.~52, prints the block for Study A or B as $500,\\ 0$ over $0,\\ 514$, and the block for List 1 as $517,\\ 0$ over $0,\\ 497$.", "corrected": "``Consistent with our randomization procedure, each cell \\hl{crossing levels of distinct randomized factors} contains approximately one-quarter of the subjects\\hl{; mutually exclusive levels of the same factor have structural zeros}.''", "intended_meaning": "Table 1 crosses six randomized factors: the study (A or B) and the treatment or control condition of each of the five lists. A cell that crosses levels of two different factors holds about one-quarter of the $n=1{,}014$ subjects; these counts range from 219 to 278. This is the intended statement, and it is correct. A block that crosses a factor with itself holds the two marginal counts on its diagonal and two zeros beside it, since no subject is in both levels of one factor. The printed exception does not cover these twelve zeros, unless ``the diagonal'' is read as the six diagonal blocks. All 84 counts of Table 1 agree with a recalculation from the replication files." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-E011", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "complete_derivation", "exhibits": [ { "alt": "Crop of Proposition 3. A red box marks 'Assumptions 1-4' in 'Under the null hypothesis that Assumptions 1-4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, beta = 1.' The next sentence begins 'For large n, if Assumption 4 (Non-Degenerate Distributions) holds', and is followed by the p-value formula, 2 Phi of minus the absolute value of beta-hat minus 1 over the square root of the estimated variance of beta-hat.", "caption": "Proposition 3, p. 50 (PDF p. 8). The marked range is followed by the names of Assumptions 1–3, and Assumption 4 is invoked separately in the next sentence.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N001.png" }, { "alt": "Crop of the paragraph that begins 'The results of Placebo Test I for Study B are presented in table 7.' Two red boxes mark 'As-' at the end of one line and 'sumptions 1-4' at the start of the next, in the words 'should be equal to 1, under Assumptions 1-4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence).'", "caption": "Section 5.3, paragraph on Table 7, p. 56 (PDF p. 14). The same range, broken across two lines.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N001-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T12", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T12" }, { "id": "P019-T21", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T21" } ], "headline": "“Assumptions 1–4” is printed twice where the four names in parentheses are those of Assumptions 1–3.", "id": "P019-N001", "location": "Proposition 3, p. 50 (PDF p. 8); Section 5.3, paragraph on Table 7, p. 56 (PDF p. 14)", "pdf_page": 8, "printed_page": "50", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T12", "P019-T21" ], "text": { "as_printed": "p.~50: ``Proposition 3. Under the null hypothesis that \\hl{Assumptions 1--4} (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, $\\beta=1$. For large $n$, if Assumption 4 (Non-Degenerate Distributions) holds, then a two-sided p-value is given by\n\\[\n2\\Phi\\left(\\frac{-|\\hat\\beta-1|}{\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\beta]}}\\right),\n\\]\nwhere $\\Phi(\\cdot)$ is the normal CDF.''\n\np.~56: ``Among the subgroup of respondents who answer ``Yes'' to the direct question, the list experiment difference-in-means estimate $\\beta$ should be equal to 1, under \\hl{Assumptions 1--4} (Monotonicity, No Liars, No Design Effects, and Treatment Independence).''", "corrected": "p.~50: ``Proposition 3. Under the null hypothesis that \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, and Treatment Independence) hold, \\hl{provided $\\Pr[Z_i=z,Y_i=1]>0$ for $z=0,1$}, $\\beta=1$. For large $n$, if Assumption 4 (Non-Degenerate Distributions) holds, then a two-sided p-value is given by\n\\[\n2\\Phi\\left(\\frac{-|\\hat\\beta-1|}{\\sqrt{\\widehat{\\operatorname{Var}}[\\hat\\beta]}}\\right),\n\\]\nwhere $\\Phi(\\cdot)$ is the normal CDF.''\n\np.~56: ``\\ldots\\ should be equal to 1, under \\hl{Assumptions 1--3} (Monotonicity, No Liars, No Design Effects, and Treatment Independence).''", "evidence": "The evidence is a check of the two passages against the definitions of the assumptions on pp.~46--47 and against p.~2 of the supplement, and a derivation of $\\beta=1$ from Assumptions 1--3. Among respondents with $Y_i=1$, Assumption 1 implies that $X_i=1$, Assumption 2 then gives $V_i=W_i+Z_i$, and by Assumption 3 the mean of $W_i$ is the same in the two arms.", "problem": "Monotonicity is Assumption 1, No Liars and No Design Effects together are Assumption 2, and Treatment Independence is Assumption 3. Assumption 4 is Non-Degenerate Distributions, which the second sentence of Proposition 3 invokes separately. The range that matches the four names is 1--3. The first paragraph of Section 4.1 lists the same four names, and Appendix C of the supplement calls the test ``a joint test of Assumptions 1-3''. The printed implication is not false, since $\\beta=1$ also holds under the stronger hypothesis. The misprint is in the label of the null hypothesis.", "what_survives": "Nothing depends on the range. The identity $\\beta=1$ and the p-value for large samples are unchanged. Assumption 4 is still needed, as the proposition says, so that $\\beta$ is defined and the normal approximation applies." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N001", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of a paragraph of Section 5.3. A red box, which runs over two lines, marks the clause 'a joint test via Fisher's method is insignificant as well'. The preceding sentence says that the list experiment difference-in-means estimate beta should be equal to 1 under Assumptions 1 to 4.", "caption": "Section 5.3, p. 56 (PDF p. 14). The marked clause reports the joint test; the sentence before it names the assumptions, each of which concerns one list and its direct question.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N003.png" }, { "alt": "Crop of Table 7, Study B (Lists First): Placebo Test I, with columns for the estimate beta-hat, SE, p-value and n, and five rows from Nuclear power to CNN. A red box marks the p-value column: 0.294, 0.339, 0.309, 0.293 and 0.256.", "caption": "Table 7, p. 57 (PDF p. 15). The marked column holds the five p-values that Fisher’s method combines.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N003-b.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T9", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T9" }, { "id": "P019-T10", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T10" } ], "headline": "The usual chi-square calibration of Fisher’s joint test of the five placebo tests of Study B assumes that the five placebo statistics are asymptotically independent; Assumptions 1–4, which are stated list by list, do not imply this, and the premise should be stated. The joint null is not rejected by either Fisher’s method with this premise or a joint Wald test that allows dependence.", "id": "P019-N003", "location": "Section 5.3, p. 56 (PDF p. 14), paragraph beginning “The results of Placebo Test I for Study B”; Table 7, p. 57 (PDF p. 15)", "pdf_page": 14, "printed_page": "56", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T10", "P019-T9" ], "text": { "as_printed": "``Among the subgroup of respondents who answer ``Yes'' to the direct question, the list experiment difference-in-means estimate $\\beta$ should be equal to 1, under Assumptions 1--4 (Monotonicity, No Liars, No Design Effects, and Treatment Independence). None of the values of $\\beta$ are statistically significantly different from 1 using the placebo test, and \\hl{a joint test via Fisher's method is insignificant as well}.''\n\nTable 7, p.~57, gives the five p-values 0.294, 0.339, 0.309, 0.293 and 0.256.", "corrected": "``None of the values of $\\beta$ are statistically significantly different from 1 using the placebo test. \\hl{Assuming that the five placebo statistics are asymptotically independent}, Fisher's method gives a joint statistic of \\hl{12.143 on 10 degrees of freedom} ($\\hl{p=0.276}$) and does not reject the common null.''\n\nThe statement behind this sentence, with its hypotheses, is the following. Suppose that the respondents are independent and identically distributed, that Assumptions 1--4 hold for each of the five lists, that the assignments $Z_1,\\dots,Z_5$ are mutually independent, and that $(Z_1,\\dots,Z_5)$ is independent of $(W_j,X_j,Y_j)_{j=1}^{5}$. Then $T=-2\\sum_{j=1}^{5}\\log p_j$ converges in distribution to $\\chi^2_{10}$.\n\nWithout the added premise, the sentence can report the joint Wald test instead: ``\\ldots and a joint Wald test of $\\beta_1=\\dots=\\beta_5=1$ that allows covariance between the five statistics does not reject ($W=5.147$ on 5 degrees of freedom, $p=0.398$).'' This test requires independent and identically distributed respondents and a nonsingular limiting covariance matrix.", "evidence": "The evidence is a derivation, an exact counterexample and a recalculation.\n\n\\emph{Derivation.} I index the lists by $j$ and write $q_j=\\Pr(Y_j=1)$, $\\gamma_j=\\Pr(Z_j=1)$ and $m_j=E[W_j\\mid Y_j=1]$. Under Assumptions 1--4 for list $j$, I have $\\beta_j=1$ and\n\\[\n\\sqrt n\\,(\\hat\\beta_j-1)=n^{-1/2}\\sum_{i=1}^{n}\\psi_{ij}+o_p(1),\\qquad\n\\psi_j=\\frac{Y_j}{q_j}\\cdot\\frac{Z_j-\\gamma_j}{\\gamma_j(1-\\gamma_j)}\\cdot(W_j-m_j).\n\\]\nThe five statistics are jointly asymptotically normal with covariances $E[\\psi_j\\psi_k]$. Suppose that $Z_1,\\dots,Z_5$ are mutually independent and that $(Z_1,\\dots,Z_5)$ is independent of $L=(W_j,X_j,Y_j)_{j=1}^{5}$. For $j\\neq k$, the conditional expectation $E[\\psi_j\\psi_k\\mid L]$ contains the factor $E[Z_j-\\gamma_j]\\,E[Z_k-\\gamma_k]=0$. Then the limiting covariance matrix is diagonal, the five p-values are asymptotically independent and uniform, and $T$ converges in distribution to $\\chi^2_{10}$.\n\n\\emph{Counterexample.} For each respondent, let $D\\in\\{0,1\\}$, $U\\in\\{-1,1\\}$ and $Z_1,\\dots,Z_5\\in\\{0,1\\}$ be independent, each taking its two values with equal probability. Let $A_j=2Z_j-1$ and define, with empty products equal to 1,\n\\[\nX_j=D,\\qquad W_j=\\tfrac32+\\tfrac12\\,U\\prod_{kj}A_k=1\\Bigr\\},\\qquad V_j=W_j+DZ_j .\n\\]\nThe baseline response to list $j$ depends on the assignments to the earlier lists only, and the direct answers, which follow all the lists, depend on the assignments to the later lists; the order of Study B permits both. Every list satisfies Assumptions 1--4 with respect to its own assignment, and $\\beta_j=1$ for every $j$. Enumerating the 128 equally likely outcomes in exact arithmetic, I find that the covariance matrix of $(\\psi_1,\\dots,\\psi_5)$ is\n\\[\n\\begin{pmatrix}4&2&2&2&2\\\\2&4&2&2&2\\\\2&2&4&2&2\\\\2&2&2&4&2\\\\2&2&2&2&2\\end{pmatrix}.\n\\]\nThe standardized statistics have correlation $1/2$ among the first four lists and $1/\\sqrt2$ between each of these and the fifth, while each individual p-value remains valid. In this limit the variance of $T$ exceeds 20, and a $\\chi^2_{10}$ variable has variance exactly 20. Hence $T$ does not converge to $\\chi^2_{10}$. The example is artificial. It shows that the premise does not follow from the stated assumptions, and it does not show that such dependence is present in the data.\n\n\\emph{Recalculation.} From the replication files, I recover the entries of Table 7. The two joint tests are as follows.\n\\begin{audittable}{lrrrl}{Joint tests of $\\beta_1=\\dots=\\beta_5=1$ in Study B, from the replication files.}\nTest & Statistic & Degrees of freedom & p-value & Requires \\\\\nFisher's method & 12.143 & 10 & 0.276 & asymptotically independent statistics \\\\\nJoint Wald test & 5.147 & 5 & 0.398 & nonsingular limiting covariance \\\\\n\\end{audittable}\nThe Wald statistic is $W=(\\hat\\beta-\\mathbf 1)^{\\top}\\widehat C^{-1}(\\hat\\beta-\\mathbf 1)$, where $\\widehat C$ is the covariance matrix of $\\hat\\beta$ estimated from the respondents' influence values, so that covariance between lists is allowed. The estimated correlations between lists lie between $-0.034$ and $0.069$.", "problem": "Fisher's method combines the five p-values $p_1,\\dots,p_5$ of Table 7 into $T=-2\\sum_{j=1}^{5}\\log p_j$ and refers $T$ to the chi-square distribution with 10 degrees of freedom. This reference distribution is correct when the five p-values are independent, which here requires the five placebo statistics $\\hat\\beta_j$ to be asymptotically independent. The paper does not state this premise, and Assumptions 1--4 do not supply it. Each assumption concerns one list experiment and its direct question. In particular, Treatment Independence makes $(W_i,X_i,Y_i)$ for a topic independent of the assignment $Z_i$ to the list on that topic, and it says nothing about the assignments to the other four lists. In Study B each respondent answered all five lists and then the five direct questions. Therefore the assignment to one list may affect the baseline response to a later list, or a later direct answer, without violating any of the assumptions as stated; Section 5.3 itself raises the possibility of such priming. Note that the calibration does not fail merely because the same respondents enter all five tests. If the five assignments are mutually independent and jointly independent of the full vector $(W_j,X_j,Y_j)_{j=1}^{5}$, then the five statistics are asymptotically independent. The paper states no such joint condition.", "what_survives": "The entries of Table 7, the five individual tests, the value of Fisher's statistic and its nonrejection all stand, and a joint test that allows dependence between the lists does not reject either. No other result of the paper uses this step. The joint p-value from Fisher's method rests on a premise beyond Assumptions 1--4, and the paper's own discussion of priming and order effects shows that the design does not guarantee this premise." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N003", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" }, { "before_review": null, "class": "error", "date_modified": "2026-10-25", "date_published": "2026-10-25", "evidence_type": "exact_counterexample", "exhibits": [ { "alt": "Crop of the heading 'Appendix C: Placebo Test I Power Analysis' and the first seven lines below it. A red box marks 'it must either be because' in the sentence 'If this estimate is different from 1.0, it must either be because a) some of those who answer Yes are falsely confessing (thereby violating monotonicity) or b) the standard list experiment assumptions of No Liars and No Design Effects are not met.' The first sentence of the paragraph calls the test 'a joint test of Assumptions 1-3 (Monotonicity, No Liars, No Design Effects, and Treatment Independence)'.", "caption": "Online supplement, Appendix C, first paragraph, p. 2. The marked words present the two causes that follow as the only ones.", "url": "https://pmaronow.github.io/llm-audit/works/P019/exhibits/P019-N006.png" } ], "formal_release": null, "formal_statements": [ { "id": "P019-T7", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T7" }, { "id": "P019-T8", "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T8" } ], "headline": "A list difference other than 1 among respondents who answer “Yes” can also come from a failure of Treatment Independence, which the supplement’s list of two causes leaves out.", "id": "P019-N006", "location": "Online supplement, Appendix C, first paragraph, third sentence, p. 2 (supplement PDF p. 2). The related passage in the main text is Section 4.1 and Proposition 3, p. 50 (PDF p. 8)", "pdf_page": 2, "printed_page": "2", "reaches": [], "related_entries": [], "revisions": [], "same_defect_as": [], "severity": "Minor", "subject": "paper", "target_ids": [ "P019-T7", "P019-T8" ], "text": { "as_printed": "``The placebo test proposed in Section 4 is a joint test of Assumptions 1-3 (Monotonicity, No Liars, No Design Effects, and Treatment Independence). In brief, the test considers whether the conventional list experimental estimate appears to be significantly different from 1.0 among the subset of subjects who answer ``Yes'' to the direct question. If this estimate is different from 1.0, \\hl{it must either be because} a) some of those who answer ``Yes'' are falsely confessing (thereby violating monotonicity) or b) the standard list experiment assumptions of No Liars and No Design Effects are not met.''", "corrected": "``If \\hl{the population difference-in-means among subjects who answer ``Yes''} is different from 1.0, \\hl{at least one of Assumptions 1-3 must fail}: some subjects falsely confess, No Liars or No Design Effects fails, \\hl{or Treatment Independence fails. A statistically significant sample departure is evidence against this joint null.}''\n\nFor a population difference, the two-cause explanation is also correct if it begins ``Provided Assumption 3 (Treatment Independence) holds''. That version covers less, and the paper reports likely violations of Assumption 3 in Study B.", "evidence": "The evidence is a derivation and an exact counterexample. Under Assumptions 1 and 2, every respondent with $Y_i=1$ has $X_i=1$ and $V_i=W_i+Z_i$, and I have\n\\[\n\\beta=1+\\mathrm{E}[W_i\\mid Z_i=1,Y_i=1]-\\mathrm{E}[W_i\\mid Z_i=0,Y_i=1].\n\\]\nAssumption 3 implies that the last two terms are equal, and without it they need not be. For the counterexample, let $B$, $C$, $X$ and $Z$ be independent Bernoulli variables with mean $1/2$, and set\n\\[\nW=B+C,\\qquad Y=X\\,\\mathbf{1}\\{C=Z\\},\\qquad V=W+XZ.\n\\]\nSince $Y\\le X$, Assumption 1 holds, and the three classes of respondents $(X,Y)=(0,0)$, $(1,0)$ and $(1,1)$ have probabilities $1/2$, $1/4$ and $1/4$. Assumption 2 holds by the definition of $V$. The treatment $Z$ is independent of $(B,C,X)$, but the direct answer depends on $Z$, and Assumption 3 fails: the event $(W,X,Y)=(0,1,0)$ has probability $1/16$ and never occurs together with $Z=0$. The list response $V$ has the following moments in the four cells.\n\\begin{audittable}{rrrrr}{The counterexample: probability of each cell of $(Z,Y)$, and the mean and variance of the list response $V$ in the cell.}\n$Z$ & $Y$ & Probability & $\\mathrm{E}[V\\mid Z,Y]$ & $\\operatorname{Var}[V\\mid Z,Y]$ \\\\\n\\hline\n0 & 0 & $3/8$ & $7/6$ & $17/36$ \\\\\n0 & 1 & $1/8$ & $1/2$ & $1/4$ \\\\\n1 & 0 & $3/8$ & $7/6$ & $17/36$ \\\\\n1 & 1 & $1/8$ & $5/2$ & $1/4$ \\\\\n\\end{audittable}\nAssumption 4 holds, since all four variances are positive, $\\operatorname{Var}[Y]=3/16$ and $\\operatorname{Var}[Z]=1/4$. From the two rows with $Y=1$, I obtain $\\beta=5/2-1/2=2$. The share of ``Yes'' answers is $1/4$ in each arm, so that $\\delta=0$. Equal response rates in the two arms therefore do not imply Assumption 3.", "problem": "The sentence presents two causes as the only ones: false confessions, which violate Assumption 1, or a failure of No Liars or No Design Effects, which together are Assumption 2. The joint null of the test also contains Treatment Independence (Assumption 3), as the first sentence of the paragraph and Proposition 3 say. A failure of Assumption 3 alone can make the population difference $\\beta$ differ from 1. This happens when the answer to the direct question depends on the list treatment. Then the respondents who answer ``Yes'' in the two arms differ in their baseline list responses, although no one confesses falsely, no one lies on the list, and the responses to the non-sensitive items do not change. Note that an effect of the treatment on the direct answer is not a design effect in the sense of Assumption 2; the paper places it under Assumption 3 (p.~47). Here I read ``estimate'' as the population quantity $\\beta$, so that sampling variation is not at issue.", "what_survives": "Section 4.1 and Proposition 3 include Treatment Independence in the null, and the implication $\\beta=1$ under the stated assumptions is unaffected. In the simulations of Appendix C, the direct answers are fixed before the treatment is assigned at random. Assumption 3 holds there, and the omitted cause does not arise. The power values were not computed again for this finding. The application already separates Study A, in which the direct questions come first, from Study B, for which the paper discusses failures of Treatment Independence. What changes is the diagnosis. A rejection by Placebo Test I cannot in general be attributed only to false confessions or to the list responses." }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-N006", "work": { "arxiv_id": "1312.1268", "doi": "10.1093/jssam/smu023", "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "url": "https://pmaronow.github.io/llm-audit/works/P019/" }, "work_id": "P019" } ], "lean": { "applicable": true, "guide": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/docs/works/P019.md", "repository": "https://github.com/pmaronow/llm-audit-lean", "targets": [ { "declarations": [ { "claim_revision_id": "catalog:90e99c0dff6a07aada105fd871e39af3102254d3dc30a283576e6184b760a2a1", "conditional_on": [], "credit_reason": "A mechanism-mismatch witness; no exact assertion about Monte Carlo power is credited to this row. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ceiling_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L602", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:90e99c0dff6a07aada105fd871e39af3102254d3dc30a283576e6184b760a2a1", "conditional_on": [], "credit_reason": "Auxiliary baseline fact, not a standalone counterexample or power result. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.binomial_baseline_exact", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L572", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:90e99c0dff6a07aada105fd871e39af3102254d3dc30a283576e6184b760a2a1", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T1", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T1" }, "html": { "body": "

supplement Appendix C, pp. 2–3, Figure A1 lower-left panel. The printed and coded ceiling mechanisms have different response vectors and exact mixture contrasts.

", "headline": "Supplement Appendix C, the simulated design effect (“ceiling effect”) and the design-effect clause of the 80% power sentence, as printed; Figure A1, lower-left panel", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph and footnote 1; p. 3, first paragraph and Figure A1, lower-left panel 'Violated assumption: no design effects'. Not in main.pdf; the main-text counterpart is Section 4.1, last paragraph, pp. 50–51 (PDF pp. 8–9)" }, "id": "P019-T1", "text": { "body": "supplement Appendix C, pp. 2–3, Figure A1 lower-left panel. The printed and coded ceiling mechanisms have different response vectors and exact mixture contrasts.", "headline": "Supplement Appendix C, the simulated design effect (“ceiling effect”) and the design-effect clause of the 80% power sentence, as printed; Figure A1, lower-left panel", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph and footnote 1; p. 3, first paragraph and Figure A1, lower-left panel 'Violated assumption: no design effects'. Not in main.pdf; the main-text counterpart is Section 4.1, last paragraph, pp. 50–51 (PDF pp. 8–9)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T1" }, { "declarations": [ { "claim_revision_id": "catalog:686cf803b7381c5115a3fbe968fe02d47e8dcb722c7f191c6aa80b41c090d84f", "conditional_on": [], "credit_reason": "This row covers the exact response/contrast component, not the empirical about-80%-power clause retained in the whole target text. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ceiling_corrected", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L614", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:686cf803b7381c5115a3fbe968fe02d47e8dcb722c7f191c6aa80b41c090d84f", "conditional_on": [], "credit_reason": "The pointwise mechanism component is stated without any unnecessary distribution hypothesis. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ceiling_response_increments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L636", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:686cf803b7381c5115a3fbe968fe02d47e8dcb722c7f191c6aa80b41c090d84f", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T2", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T2" }, "html": { "body": "

supplement Appendix C, pp. 2–3, Figure A1 lower-left panel. The corrected ceiling response has the stated subject increments and both exact mixture-contrast formulas.

", "headline": "Supplement Appendix C, description of the simulated design effect and the design-effect clause of the 80% power sentence, corrected", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph; p. 3, first paragraph and Figure A1, lower-left panel" }, "id": "P019-T2", "text": { "body": "supplement Appendix C, pp. 2–3, Figure A1 lower-left panel. The corrected ceiling response has the stated subject increments and both exact mixture-contrast formulas.", "headline": "Supplement Appendix C, description of the simulated design effect and the design-effect clause of the 80% power sentence, corrected", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph; p. 3, first paragraph and Figure A1, lower-left panel" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T2" }, { "declarations": [ { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/P019_Audit.lean:692 proves all four standing assumptions for the stated witness, empty-cell probabilities 5/8 at n=1, and covariances 5/64 and 3/64. Target T3(i) explicitly distinguishes the undefined printed ratios from the numerical covariances obtained if empty subgroup means are given value zero; singletonMean at P019/P019_Audit.lean:686 implements exactly that completion. Neither the claim component nor the row requires failure for every possible completion, so the reviewer's limitation is not a material fidelity problem.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.lemma_two_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L692", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/P019_Audit.lean:716 proves, in the valid witness population, P(Z=1 | Y=0)=1/2, the zero-completed conditional subgroup mean 5/12, and the population cell mean 5/6. Target T3(ii)'s omitted-count-mass discrepancy already occurs at k=1, and T3 explicitly supplies the zero completion for its numerical illustrations. The printed ratio's undefined empty-cell value limits the interpretation of the illustration, but introduces no discrepancy from this target component.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.conditional_count_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L716", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "conditional_on": [], "credit_reason": "Only the arithmetic support component is credited; there is no probabilistic binomial-sum theorem in this declaration. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.omitted_binomial_mass", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L706", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "conditional_on": [], "credit_reason": "The positive mass at count zero suffices to refute an almost-sure equality with n gamma=2. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.realized_count_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1407", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/Wave1/Gap/GP019B4b.lean:733 uses the specified fair-bit population and proves the two zero covariances conditional on D, the unequal product-variance values 141205/404496 and 75809/202248, and the treatment-count probabilities 3/8 and 1/16. Target T3(iii) expressly says that every moment in this calculation is conditional on D, and T3(iv) specifies these count facts. The conditional calculation therefore matches this row's component; the target does not ask this component to refute every unconditional completion or Proposition 1.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.finite_product_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B4b.lean#L732", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f487fd991afdaf1a9aaae0e720180e8b3bd456ad46397abfd5c0f695a32000bd", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T3", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T3" }, "html": { "body": "

Lemma 2, p. 48; Appendix B, pp. 62–63, displays (6), (9), and the following sentence. The four independent fair-bit population has the complete stated cell and observable laws. Its printed ratios have positive empty-cell probabilities for every sample size; zero completion gives the stated nonzero covariances. Conditional binomial counts expose the missing mass for every k. At n=4 the six occupied-cell moments refute the exact product-variance formula, and the realized treatment count need not equal its expectation. The comparison uses the specified completion and conditioning.

", "headline": "Lemma 2 and the finite-sample steps of its proof and of the proof of Proposition 1 (display (6), the first equality of (9), and the sentence after (9)), as printed", "location": "Lemma 2: p. 48 (PDF p. 6). Appendix B.1: proof of Lemma 2, displays (4)–(6), p. 62 (PDF p. 20), and (7), p. 63 (PDF p. 21); proof of Proposition 1, displays (8)–(9) and the sentence after (9), p. 63 (PDF p. 21)" }, "id": "P019-T3", "text": { "body": "Lemma 2, p. 48; Appendix B, pp. 62–63, displays (6), (9), and the following sentence. The four independent fair-bit population has the complete stated cell and observable laws. Its printed ratios have positive empty-cell probabilities for every sample size; zero completion gives the stated nonzero covariances. Conditional binomial counts expose the missing mass for every k. At n=4 the six occupied-cell moments refute the exact product-variance formula, and the realized treatment count need not equal its expectation. The comparison uses the specified completion and conditioning.", "headline": "Lemma 2 and the finite-sample steps of its proof and of the proof of Proposition 1 (display (6), the first equality of (9), and the sentence after (9)), as printed", "location": "Lemma 2: p. 48 (PDF p. 6). Appendix B.1: proof of Lemma 2, displays (4)–(6), p. 62 (PDF p. 20), and (7), p. 63 (PDF p. 21); proof of Proposition 1, displays (8)–(9) and the sentence after (9), p. 63 (PDF p. 21)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T3" }, { "declarations": [ { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "This row supplies the equality; strict positivity under the maintained model is separately exported by R129-10. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.empty_cell_probability", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1033", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/Wave1/Gap/GLP019B3.lean:716 exports the empty-cell probabilities, conditional binomial probabilities, occupied-pattern cell moments, and the corrected zero covariances specified by T4(a,b) and this row's pattern-moment component. Its positive-probability guards give conditioning its ordinary meaning, and n>=2 is necessary for P(D)>0. The n=0,1 count endpoints and counts outside the binomial support are mathematically trivial; finite weighted laws preserve the entire joint distribution of fixed finite-list counts and binary indicators.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.corrected_cell_lemmas_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GLP019B3.lean#L706", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/Wave1/Gap/GP019B4.lean:677 states E[cellMean | D,A=a]=m_z, the occupied-pattern contrast mean b and variance h, and the two nonempty-count probabilities 1-(1-gamma)^k and 1-gamma^k. These are exactly the conditional-mean, contrast-moment, and count-endpoint components in T4(a-c). The guards exclude only null conditioning events, while n>=2 removes the impossible occupied-cell sizes and trivial count endpoints; neither the fixed completion constants nor the finite observable-law representation loses a material case.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.corrected_conditional_means_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B4.lean#L675", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/Wave1/Gap/GP019B4.lean:713 gives both Var(muHat | D) and Var(AB | D) as b^2 Var(A | D)+E[A^2 h | D], and states that the discrepancy from the printed three-term expression is Cov(A^2,h | D). Those are the exact corrected identities in T4(c,d); expanding the covariance also gives the target's intermediate expression b^2 Var(A | D)+E[A^2 | D]E[h | D]. The n>=2 condition is the domain where D can have positive probability, and the arbitrary fixed empty-cell values disappear on D.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.corrected_product_variance_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B4.lean#L707", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/Wave1/Gap/GLP019B3.lean:1230 defines R_n as Cov(A^2,h | D) under the full n-fold IID law and concludes both R_n=O(n^-2) and n^2 R_n tending to -2 mu p (s_1/gamma+s_0/(1-gamma)). This matches the covariance-order and leading-coefficient component of corrected T4(d) on the entire natural sample-size sequence. Totalized conditional moments at n=0,1 change only finitely many terms, and finite support represents every relevant bounded-list population law.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.corrected_covariance_order_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GLP019B3.lean#L1218", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "The complete covariance-order declaration already supplies the actual rate and limit coefficient.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.covariance_taylor_coefficient", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1375", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "The distinction between expected and realized counts is exact. n=0 contributes only zero identities and no substantive omitted case. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.treatment_count_correction", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1388", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "conditional_on": [], "credit_reason": "Not invoked separately: the retained conditional moments are supplied by the composite cell and conditional-mean declarations.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.conditional_sample_factorization", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1730", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:31231d2502b5a1a9dec1d2f595bef3fd9874c58ff7f07103a08f8a43620ba2c2", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T4", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T4" }, "html": { "body": "

corrected Lemma 2, p. 48, and Appendix B, pp. 62–63, displays (6)–(9). For iid observations from any bounded count-valued population satisfying Assumptions 1–4, the empty-cell probabilities, conditional binomial law and missing-mass sums hold for every n, including zero and one. K=n(1-Ybar). On occupied cells, the stated pattern-conditional means, variances and covariances give the corrected Lemma 2 and exact product-variance identity. Its discrepancy from the printed product expression is Cov(A²,h | D), of order n⁻² with the stated limit coefficient. The unconditional direct-response moments and population contrast identities hold, and n gamma is the expected treatment count; the realized count is n gammaHat. Conditional moments are asserted only on positive-probability events.

", "headline": "Lemma 2, the first equality of (9) and the sentence after (9), corrected", "location": "Lemma 2: p. 48 (PDF p. 6). Proof of Lemma 2, displays (4)–(7): pp. 62–63 (PDF pp. 20–21). Proof of Proposition 1, displays (8)–(9) and the sentence after (9): p. 63 (PDF p. 21)" }, "id": "P019-T4", "text": { "body": "corrected Lemma 2, p. 48, and Appendix B, pp. 62–63, displays (6)–(9). For iid observations from any bounded count-valued population satisfying Assumptions 1–4, the empty-cell probabilities, conditional binomial law and missing-mass sums hold for every n, including zero and one. K=n(1-Ybar). On occupied cells, the stated pattern-conditional means, variances and covariances give the corrected Lemma 2 and exact product-variance identity. Its discrepancy from the printed product expression is Cov(A²,h | D), of order n⁻² with the stated limit coefficient. The unconditional direct-response moments and population contrast identities hold, and n gamma is the expected treatment count; the realized count is n gammaHat. Conditional moments are asserted only on positive-probability events.", "headline": "Lemma 2, the first equality of (9) and the sentence after (9), corrected", "location": "Lemma 2: p. 48 (PDF p. 6). Proof of Lemma 2, displays (4)–(7): pp. 62–63 (PDF pp. 20–21). Proof of Proposition 1, displays (8)–(9) and the sentence after (9): p. 63 (PDF p. 21)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T4" }, { "declarations": [ { "claim_revision_id": "catalog:e907bb114c10b0f27c079c6176a9cd5900bb594d29dcfb5c680dcb8ebf744f07", "conditional_on": [], "credit_reason": "The target of the counterexample is the prose comparison with treated V+1; the preceding displayed identity with V=t+1 is correct. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.distribution_sign_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L759", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:e907bb114c10b0f27c079c6176a9cd5900bb594d29dcfb5c680dcb8ebf744f07", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T5", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T5" }, "html": { "body": "

Section 4.1, p. 50, final distribution-comparison sentence. The independent fair X,W,Z population with Y=X satisfies all four assumptions. Every cell and marginal variance is positive as specified. The full conditional laws of control V and treated V+1 have disjoint supports {0,1} and {2,3}. The preceding displayed identity is true for every real t: control V has the same conditional law as treated V-1, including equality of all distribution functions.

", "headline": "Section 4.1 (Placebo Test I), the sentence on tests of distributional equality, as printed", "location": "p. 50 (PDF p. 8), Section 4.1, first paragraph" }, "id": "P019-T5", "text": { "body": "Section 4.1, p. 50, final distribution-comparison sentence. The independent fair X,W,Z population with Y=X satisfies all four assumptions. Every cell and marginal variance is positive as specified. The full conditional laws of control V and treated V+1 have disjoint supports {0,1} and {2,3}. The preceding displayed identity is true for every real t: control V has the same conditional law as treated V-1, including equality of all distribution functions.", "headline": "Section 4.1 (Placebo Test I), the sentence on tests of distributional equality, as printed", "location": "p. 50 (PDF p. 8), Section 4.1, first paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T5" }, { "declarations": [ { "claim_revision_id": "catalog:faebad18b4aab59c28050c73476d3272651ecacf6164f00d06e57992b6d13e36", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/P019_Audit.lean:447 assumes precisely monotonicity, the response equation, treatment independence, and nonzero confessor-cell probabilities, and equates each real point mass of control V and treated V-1 to that of W given Y=1. These are the hypotheses and conclusion explicitly printed in corrected target T6's formal content. Since all these counts have finite observable support, equality at every real point is equality of the full conditional laws; nonnegative masses make nonzero event probability equivalent to positive probability.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_distribution_corrected", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L447", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:faebad18b4aab59c28050c73476d3272651ecacf6164f00d06e57992b6d13e36", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T6", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T6" }, "html": { "body": "

corrected Section 4.1 distribution-comparison sentence, p. 50. Under Assumptions 1–3 and positive affirmative-cell probabilities, control V and treated V-1 have identical conditional point probabilities, both equal to the baseline law given an affirmative response.

", "headline": "Section 4.1 (Placebo Test I), the sentence on tests of distributional equality, corrected", "location": "p. 50 (PDF p. 8), Section 4.1, first paragraph" }, "id": "P019-T6", "text": { "body": "corrected Section 4.1 distribution-comparison sentence, p. 50. Under Assumptions 1–3 and positive affirmative-cell probabilities, control V and treated V-1 have identical conditional point probabilities, both equal to the baseline law given an affirmative response.", "headline": "Section 4.1 (Placebo Test I), the sentence on tests of distributional equality, corrected", "location": "p. 50 (PDF p. 8), Section 4.1, first paragraph" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T6" }, { "declarations": [ { "claim_revision_id": "catalog:030e6fdbf6a978489b268646a0902dca43854dd2fdce23660ee65961b0ea071e", "conditional_on": [], "credit_reason": "This is a witness to the omitted Treatment Independence cause, not a refutation of Proposition 3 under its maintained hypotheses. The target’s alternative eight-atom witness is unnecessary once this valid interior-prevalence witness is supplied. Claim-stage verdict unchanged after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.diagnosis_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L786", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:030e6fdbf6a978489b268646a0902dca43854dd2fdce23660ee65961b0ea071e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T7", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T7" }, "html": { "body": "

supplement Appendix C, p. 2, first paragraph; Section 4.1, p. 50. Both the sixteen-atom and eight-atom fair-bit constructions satisfy Monotonicity, No Liars and No Design Effects, and Non-Degeneracy, with delta zero and beta two. The first has the stated latent classes, cell moments, potential-response independence, marginal Y independence, and explicit zero-versus-1/32 failure of whole-triple independence. Thus Treatment Independence supplies a missing cause of beta being different from one, while false confession and list design effects are absent in both examples.

", "headline": "Supplement Appendix C, first paragraph, the sentence 'If this estimate is different from 1.0, it must either be because ...', as printed", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, first paragraph, third sentence. Not in main.pdf; the main-text counterpart is Section 4.1 and Proposition 3, p. 50 (PDF p. 8)" }, "id": "P019-T7", "text": { "body": "supplement Appendix C, p. 2, first paragraph; Section 4.1, p. 50. Both the sixteen-atom and eight-atom fair-bit constructions satisfy Monotonicity, No Liars and No Design Effects, and Non-Degeneracy, with delta zero and beta two. The first has the stated latent classes, cell moments, potential-response independence, marginal Y independence, and explicit zero-versus-1/32 failure of whole-triple independence. Thus Treatment Independence supplies a missing cause of beta being different from one, while false confession and list design effects are absent in both examples.", "headline": "Supplement Appendix C, first paragraph, the sentence 'If this estimate is different from 1.0, it must either be because ...', as printed", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, first paragraph, third sentence. Not in main.pdf; the main-text counterpart is Section 4.1 and Proposition 3, p. 50 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T7" }, { "declarations": [ { "claim_revision_id": "catalog:dee57a5f6e020592a359b9bd59e7f936047798eced5d3679a9eb7691ad142fbf", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/P019_Audit.lean:411 states beta=1+E[W | Z=1,Y=1]-E[W | Z=0,Y=1] under Assumptions 1-2 and defined confessor cells, exactly T8(i). It does not assume treatment independence, and its structural assumptions apply on every positive-mass atom, which is the finite-law form of almost-sure assumptions. A fixed finite list gives bounded natural counts, so pushing a population law forward to its joint count/indicator values preserves this identity and its entire domain.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.beta_baseline_decomposition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L411", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dee57a5f6e020592a359b9bd59e7f936047798eced5d3679a9eb7691ad142fbf", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/P019_Audit.lean:425 concludes beta=1 under Assumptions 1-3 and defined confessor cells, exactly the beta-equals-one component of T8(ii) identified by the row. T8(ii)'s baseline-mean equality also follows directly from this identity and the separately stated baseline decomposition at P019/P019_Audit.lean:411. The finite observable-law representation loses no fixed-list population cases, and requiring nonzero confessor cells merely makes the claim's conditional means defined.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_one_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L425", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:dee57a5f6e020592a359b9bd59e7f936047798eced5d3679a9eb7691ad142fbf", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': \"P019/P019_Audit.lean:436 assumes defined confessor cells and beta!=1 and concludes failure of monotonicity, the response equation, or treatment independence. This is exactly corrected T8(iii)'s exhaustive diagnosis, and adding Assumptions 1-2 immediately gives its final implication that Assumption 3 fails. The finite weighted representation and nonzero-event guards preserve the bounded-list population cases and ordinary conditioning required by the claim.\"}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.beta_departure_diagnosis", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L436", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:dee57a5f6e020592a359b9bd59e7f936047798eced5d3679a9eb7691ad142fbf", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T8", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T8" }, "html": { "body": "

corrected supplement Appendix C, p. 2, first paragraph. The baseline-composition decomposition yields beta one under Assumptions 1–3 and both corrected failure diagnoses when the population contrast differs from one.

", "headline": "Supplement Appendix C, first paragraph, the sentence 'If this estimate is different from 1.0, it must either be because ...', corrected", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, first paragraph, third sentence" }, "id": "P019-T8", "text": { "body": "corrected supplement Appendix C, p. 2, first paragraph. The baseline-composition decomposition yields beta one under Assumptions 1–3 and both corrected failure diagnoses when the population contrast differs from one.", "headline": "Supplement Appendix C, first paragraph, the sentence 'If this estimate is different from 1.0, it must either be because ...', corrected", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, first paragraph, third sentence" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T8" }, { "declarations": [ { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "A witness to failure of Fisher calibration under the target’s specified per-list reading, not a claim of dependence in the observed survey or a particular rejection probability at 0.05.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.fisher_reference_counterexample_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GLP019B4.lean#L899", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "Auxiliary finite-witness verification only; R130-04 supplies the actual asymptotic counterexample.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.fisher_sequential_finite_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B5b.lean#L296", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "Not invoked separately: the finite composite declaration includes whole-triple independence in its standing assumptions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.sequential_own_independence_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B5b.lean#L310", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "Not invoked separately: the finite composite declaration already proves the standing assumptions for every list.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.sequential_standing_proof", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B5b.lean#L313", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "Not invoked separately: the finite composite declaration already includes the entire influence matrix.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.sequential_influence_matrix", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1679", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "conditional_on": [], "credit_reason": "Not invoked separately: the finite composite declaration already includes centered influence functions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.sequential_influence_centered", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1697", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:1443bc6a4cec18b984ee3f89830e12bfb43b83c824eeb35b9e6458e49411f3ca", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T9", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T9" }, "html": { "body": "

Section 5.3, p. 56, the Fisher joint-test reference distribution. The seven independent fair coordinates give the stated ordered five-list construction, each list's Assumptions 1–4 and beta one, exact cell summaries, influence functions, and root-n linearization. The joint Gaussian limit has standard normal margins and correlations 1/2 and 1/sqrt(2); every marginal p-value tends to the uniform law. Fisher's limit is the stated Gaussian transform, with variance 20+12 Var(h(E5))+8 Cov(h(E5),g(E5)) greater than 20, whereas chi-square ten has variance 20. Every expectation defining h(t) is integrable, for every real t. The statistics are defined with probability tending to one and their limits hold for every completion. Thus these per-list assumptions and independent assignments do not ensure Fisher calibration.

", "headline": "Section 5.3, 'a joint test via Fisher’s method is insignificant as well' (the chi-square reference distribution of the joint test of the five Study B placebo tests), as printed", "location": "p. 56 (PDF p. 14), Section 5.3, paragraph on Table 7; Table 7 is on p. 57 (PDF p. 15)" }, "id": "P019-T9", "text": { "body": "Section 5.3, p. 56, the Fisher joint-test reference distribution. The seven independent fair coordinates give the stated ordered five-list construction, each list's Assumptions 1–4 and beta one, exact cell summaries, influence functions, and root-n linearization. The joint Gaussian limit has standard normal margins and correlations 1/2 and 1/sqrt(2); every marginal p-value tends to the uniform law. Fisher's limit is the stated Gaussian transform, with variance 20+12 Var(h(E5))+8 Cov(h(E5),g(E5)) greater than 20, whereas chi-square ten has variance 20. Every expectation defining h(t) is integrable, for every real t. The statistics are defined with probability tending to one and their limits hold for every completion. Thus these per-list assumptions and independent assignments do not ensure Fisher calibration.", "headline": "Section 5.3, 'a joint test via Fisher’s method is insignificant as well' (the chi-square reference distribution of the joint test of the five Study B placebo tests), as printed", "location": "p. 56 (PDF p. 14), Section 5.3, paragraph on Table 7; Table 7 is on p. 57 (PDF p. 15)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T9" }, { "declarations": [ { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/Wave1/Gap/GLP019B1.lean:552 concludes the five null identities, conditional and unconditional cross-influence orthogonality, joint independent normal and uniform limits, and the chi-square-10 Fisher limit. Its jointAssignmentStable hypothesis, defined at P019/P019_Audit.lean:1770, factors every latent-vector function against every assignment pattern and is equivalent to the mutual assignment independence and assignment-vector/latent-vector independence expressly required by corrected T10. Fixed zero completions and the uniform law on [0,1] change no asserted limit, and representing the bounded five-list vector by a finite law preserves all population cases covered by that target.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.fisher_corrected_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GLP019B1.lean#L537", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "conditional_on": [], "credit_reason": "Auxiliary ratio identity; the corrected Fisher declaration already supplies the statistic and Fisher limits.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ratio_residual_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1279", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "conditional_on": [], "credit_reason": "Auxiliary influence identity; the corrected Fisher declaration already supplies the conditional and unconditional cross-moment conclusions.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_influence_residual", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1884", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "conditional_on": [], "credit_reason": "Auxiliary transform result; the corrected Fisher declaration already supplies the required joint uniform limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.normal_pvalue_uniform_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B1.lean#L116", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "conditional_on": [], "credit_reason": "Auxiliary Fisher-law identity; the corrected Fisher declaration already supplies the chi-square limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.uniform_fisher_identity_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B1.lean#L282", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:552929593c27084efd690a42e41ab6855839c59a5c7e15e958bdf4ea17b3c727", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T10", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T10" }, "html": { "body": "

corrected Section 5.3, p. 56. The full joint assignment premise gives beta one, zero cross-influence moments, independent normal and uniform limits, and the chi-square ten Fisher limit.

", "headline": "Section 5.3, the joint test via Fisher’s method, corrected (with the premise it needs)", "location": "p. 56 (PDF p. 14), Section 5.3, paragraph on Table 7" }, "id": "P019-T10", "text": { "body": "corrected Section 5.3, p. 56. The full joint assignment premise gives beta one, zero cross-influence moments, independent normal and uniform limits, and the chi-square ten Fisher limit.", "headline": "Section 5.3, the joint test via Fisher’s method, corrected (with the premise it needs)", "location": "p. 56 (PDF p. 14), Section 5.3, paragraph on Table 7" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T10" }, { "declarations": [ { "claim_revision_id": "catalog:73eb2bab156ae3a514bede45a99eb70eafca840a1cceb27ac869449a28a070b0", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019_Audit.lean:465 exports both E[V|Z=1]=E[W]+μ and E[V|Z=0]=E[W], whose difference is the corrected equation (2), T11(a). Its noLiarsDesign and treatmentIndependent hypotheses express exactly Assumptions 2–3, and its nonzero arm probabilities are equivalent to the target’s 0Lemma 1 proof and equation (2), p. 48; Corollary 1, p. 49; Lemma 2 proof, p. 62; Proposition 2 proof, p. 64. For any probability space with the standing bounded count and binary observations, the corrected treatment-arm means, affirmative-response sensitive mean, and affirmative-cell list-mean shift hold under their respective stated assumptions. Under Assumptions 1-4, the printed Wald interval is defined with probability tending to one and its coverage tends to 1-alpha for every 0 < alpha < 1, independently of fixed completions outside its domain. No quantile-set premise is needed. The correction in (d) labels the negative responders and has no independent mathematical assertion.

", "headline": "Equation (2) and the last sentence of the proof of Lemma 1; Corollary 1; the sentence after display (4) in the proof of Lemma 2; the sentence 'By Assumptions 1 and 2 ...' in the proof of Proposition 2; all corrected", "location": "(a), (b): p. 48 (PDF p. 6). (c): p. 49 (PDF p. 7). (d): p. 62 (PDF p. 20). (e): p. 64 (PDF p. 22)" }, "id": "P019-T11", "text": { "body": "Lemma 1 proof and equation (2), p. 48; Corollary 1, p. 49; Lemma 2 proof, p. 62; Proposition 2 proof, p. 64. For any probability space with the standing bounded count and binary observations, the corrected treatment-arm means, affirmative-response sensitive mean, and affirmative-cell list-mean shift hold under their respective stated assumptions. Under Assumptions 1-4, the printed Wald interval is defined with probability tending to one and its coverage tends to 1-alpha for every 0 < alpha < 1, independently of fixed completions outside its domain. No quantile-set premise is needed. The correction in (d) labels the negative responders and has no independent mathematical assertion.", "headline": "Equation (2) and the last sentence of the proof of Lemma 1; Corollary 1; the sentence after display (4) in the proof of Lemma 2; the sentence 'By Assumptions 1 and 2 ...' in the proof of Proposition 2; all corrected", "location": "(a), (b): p. 48 (PDF p. 6). (c): p. 49 (PDF p. 7). (d): p. 62 (PDF p. 20). (e): p. 64 (PDF p. 22)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T11" }, { "declarations": [ { "claim_revision_id": "catalog:a667a2aec5b4bc53f93e6f230d66a9a62e0229ed0eae3e935704822728de304e", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019_Audit.lean:425 proves beta=1 under Assumptions 1–3 and the two defined confessor cells, exactly the explicitly stated first-sentence formal content of corrected Proposition 3 in T12. It does not require the positive conditional variances used for the separate large-sample p-value statement. The finite weighted observable representation adds no restriction in the paper’s fixed-list model, and this row claims only the population identity.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_one_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L425", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:a667a2aec5b4bc53f93e6f230d66a9a62e0229ed0eae3e935704822728de304e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T12", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T12" }, "html": { "body": "

Proposition 3, p. 50 (PDF p. 8), and Section 5.3, p. 56 (PDF p. 14): Assumptions 1–3 and defined confessor cells give beta = 1. With Assumption 4, the studentized estimator tends to a standard normal and the printed two-sided p-value is asymptotically uniform, for every fixed completion of undefined sample quantities. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.

", "headline": "Proposition 3, first sentence, and the same phrase in Section 5.3, with the assumption range corrected", "location": "Proposition 3: p. 50 (PDF p. 8). Section 5.3, paragraph on Table 7: p. 56 (PDF p. 14)" }, "id": "P019-T12", "text": { "body": "Proposition 3, p. 50 (PDF p. 8), and Section 5.3, p. 56 (PDF p. 14): Assumptions 1–3 and defined confessor cells give beta = 1. With Assumption 4, the studentized estimator tends to a standard normal and the printed two-sided p-value is asymptotically uniform, for every fixed completion of undefined sample quantities. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.", "headline": "Proposition 3, first sentence, and the same phrase in Section 5.3, with the assumption range corrected", "location": "Proposition 3: p. 50 (PDF p. 8). Section 5.3, paragraph on Table 7: p. 56 (PDF p. 14)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T12" }, { "declarations": [ { "claim_revision_id": "catalog:ec9ecb10ab967f2975f950a5a982f9aee56b8874db9d5587aae531dfa0c2455a", "conditional_on": [], "credit_reason": "The counterexample label refers to the target’s identification interpretation of the CNN wording. As the record notes, the empirical parenthetical is tentative and no printed mathematical theorem is refuted; the two-world identification obstruction itself is exact.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.cnn_nonidentification_counterexample", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L861", "status": "counterexample_proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:ec9ecb10ab967f2975f950a5a982f9aee56b8874db9d5587aae531dfa0c2455a", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T13", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T13" }, "html": { "body": "

Table A1, p. 61 (PDF p. 19), and Section 5.2, p. 54 (PDF p. 12): the two CNN worlds have independent type, baseline and assignment, satisfy Assumptions 1–4 for reading-or-watching, and share the complete observable law, with the stated cell and atom tables, moments, contrasts and placebos. Watching prevalence is 1/4 versus 1/2; all answers are truthful in A, whereas the reader-only class becomes withholding watchers in B. Thus no function of the observable law identifies watching throughout these worlds. Usual measure-theoretic conditions are assumed (automatic for this finite witness).

", "headline": "Appendix A.1, Table A1, row 'CNN (L5 & D5)' (list item 'read or watch' against direct question 'watch'), with the reading in Section 5.2 that the list experiments and the direct questions measure the same quantities", "location": "Table A1: p. 61 (PDF p. 19). Section 5.2: p. 54 (PDF p. 12)" }, "id": "P019-T13", "text": { "body": "Table A1, p. 61 (PDF p. 19), and Section 5.2, p. 54 (PDF p. 12): the two CNN worlds have independent type, baseline and assignment, satisfy Assumptions 1–4 for reading-or-watching, and share the complete observable law, with the stated cell and atom tables, moments, contrasts and placebos. Watching prevalence is 1/4 versus 1/2; all answers are truthful in A, whereas the reader-only class becomes withholding watchers in B. Thus no function of the observable law identifies watching throughout these worlds. Usual measure-theoretic conditions are assumed (automatic for this finite witness).", "headline": "Appendix A.1, Table A1, row 'CNN (L5 & D5)' (list item 'read or watch' against direct question 'watch'), with the reading in Section 5.2 that the list experiments and the direct questions measure the same quantities", "location": "Table A1: p. 61 (PDF p. 19). Section 5.2: p. 54 (PDF p. 12)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T13" }, { "declarations": [ { "claim_revision_id": "catalog:f5262114ad5ec9f777f4fa13ea6610644665bc8d1e18a92b9716c65fe9cbca97", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019_Audit.lean:888 assumes truthful Y=Q, X=R∨Q, the list-response equation, full (W,R,Q) independence from Z, and nondegenerate assignment and watching probabilities, matching T14. It exports the negative-answer contrast a/(1−b), both estimands b+a, their watching-prevalence discrepancy a and equality criterion, and beta=1 and delta=0. The intermediate Pr(X=1|Y=0)=a/(1−b) follows directly from Y=Q and X=R∨Q; compatible values on null atoms and finite observable support lose no distributional case.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.cnn_interpretation", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L888", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:f5262114ad5ec9f777f4fa13ea6610644665bc8d1e18a92b9716c65fe9cbca97", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T14", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T14" }, "html": { "body": "

Table A1, p. 61 (PDF p. 19), Section 5.2, p. 54 (PDF p. 12), and Tables 3 and 6, pp. 54,56: truthful CNN questions identify reading-or-watching, with excess over watching equal to the reader-only share, while both placebo restrictions hold. Restricted to finite underlying probability spaces with arbitrary real atom masses and pointwise truthful identities Y = Q and X = R or Q; the extension to arbitrary measurable spaces with almost-sure truthful identities is not stated.

", "headline": "The CNN item (Table A1, row 'CNN (L5 & D5)'): what the list and combined estimands are under the printed wording, corrected reading", "location": "Table A1: p. 61 (PDF p. 19). Section 5.2: p. 54 (PDF p. 12). Tables 3 and 6, row CNN: pp. 54 and 56 (PDF pp. 12 and 14)" }, "id": "P019-T14", "text": { "body": "Table A1, p. 61 (PDF p. 19), Section 5.2, p. 54 (PDF p. 12), and Tables 3 and 6, pp. 54,56: truthful CNN questions identify reading-or-watching, with excess over watching equal to the reader-only share, while both placebo restrictions hold. Restricted to finite underlying probability spaces with arbitrary real atom masses and pointwise truthful identities Y = Q and X = R or Q; the extension to arbitrary measurable spaces with almost-sure truthful identities is not stated.", "headline": "The CNN item (Table A1, row 'CNN (L5 & D5)'): what the list and combined estimands are under the printed wording, corrected reading", "location": "Table A1: p. 61 (PDF p. 19). Section 5.2: p. 54 (PDF p. 12). Tables 3 and 6, row CNN: pp. 54 and 56 (PDF pp. 12 and 14)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T14" }, { "declarations": [ { "claim_revision_id": "catalog:6ebaccefbc179d21cfcd1bb94dc8d3c1fd3370c05a0dce369ba035fb7161602b", "conditional_on": [], "credit_reason": "An acceptable algebraic extension, with no square roots or zero-division conventions involved. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.pooled_variance_difference", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1535", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6ebaccefbc179d21cfcd1bb94dc8d3c1fd3370c05a0dce369ba035fb7161602b", "conditional_on": [], "credit_reason": "The corrected two-sided-power row already uses the threshold bound and equality case; the Solution invokes its resulting tail theorem directly.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.planning_tail_argument", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1544", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:6ebaccefbc179d21cfcd1bb94dc8d3c1fd3370c05a0dce369ba035fb7161602b", "conditional_on": [], "credit_reason": "The target itself assumes S>0, so exclusion of endpoint pairs with zero unpooled variance is not an extra restriction; the declaration proves the analytic correction, without certifying execution of R or regeneration of figures. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.two_sided_power_corrected_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B1.lean#L430", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:6ebaccefbc179d21cfcd1bb94dc8d3c1fd3370c05a0dce369ba035fb7161602b", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T15", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T15" }, "html": { "body": "

supplement Appendix D, pp. 3–4, Figure A2, and Proposition 4, p. 51 (PDF p. 9): corrected normal planning power includes both tails, the missing tail lies in (0, alpha/2] with equality exactly at the null, and the N = 100 null example has one-tail power 1/40 and two-tail power 1/20.

", "headline": "Supplement Appendix D and Figure A2, panel 'N= 100' (power of Placebo Test II), corrected", "location": "Online supplement (author-appendix.pdf) p. 3 (Appendix D) and p. 4 (Figure A2, first panel); deposited code ACCG_list_placebo_II_power.R, line 31 against lines 62, 93 and 124. The test is Proposition 4, p. 51 (PDF p. 9)" }, "id": "P019-T15", "text": { "body": "supplement Appendix D, pp. 3–4, Figure A2, and Proposition 4, p. 51 (PDF p. 9): corrected normal planning power includes both tails, the missing tail lies in (0, alpha/2] with equality exactly at the null, and the N = 100 null example has one-tail power 1/40 and two-tail power 1/20.", "headline": "Supplement Appendix D and Figure A2, panel 'N= 100' (power of Placebo Test II), corrected", "location": "Online supplement (author-appendix.pdf) p. 3 (Appendix D) and p. 4 (Figure A2, first panel); deposited code ACCG_list_placebo_II_power.R, line 31 against lines 62, 93 and 124. The test is Proposition 4, p. 51 (PDF p. 9)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T15" }, { "declarations": [ { "claim_revision_id": "catalog:80f27e96ccf92b753b296104bd2fac2e067c5853966c5fe958e313e0f9b2a7df", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019_Audit.lean:646 proves ceilingBeta printedAffected ρ=1−(16/625)ρ for 0≤ρ≤1; lines 565–595 define the Binomial(4,2/5) baseline, the printed response min(W+1,4), and the corresponding mixture contrast. This matches the row’s exact printed-mechanism component: its decrement is Pr(affected and W=4)=ρ·16/625 under the independent affected selection encoded by the supplied simulation design. Integrating out affected status and assignment does not restrict that component, and the row makes no claim to establish the approximately 80% Monte Carlo power report.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ceiling_as_printed", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L646", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:80f27e96ccf92b753b296104bd2fac2e067c5853966c5fe958e313e0f9b2a7df", "conditional_on": [], "credit_reason": "This subject-level identity does not establish rejection power and is outside the retained ceiling population contrast.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.unchanged_response_increment", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L2271", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:80f27e96ccf92b753b296104bd2fac2e067c5853966c5fe958e313e0f9b2a7df", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T16", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T16" }, "html": { "body": "

supplement Appendix C, pp. 2–3, Figure A1, and Section 4.1, pp. 50–51 (PDF pp. 8–9): the printed ceiling mechanism has population contrast 1 - (16/625) times the independently mixed affected share. Restricted to this population contrast for a Binomial(4,2/5) baseline and independent affected status; omits the liar and false-confessor response rules, all rejection probabilities (including the approximately 80% and 30% examples), and all plotted finite-sample power curves and their dependence on sample size or violation share.

", "headline": "Supplement Appendix C: the simulated design effect (“ceiling effect”) and the 80% power sentence; Figure A1, lower-left panel 'Violated assumption: no design effects'", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph and footnote 1; p. 3, first paragraph and Figure A1. Main-text counterpart: Section 4.1, last paragraph, pp. 50–51 (PDF pp. 8–9)" }, "id": "P019-T16", "text": { "body": "supplement Appendix C, pp. 2–3, Figure A1, and Section 4.1, pp. 50–51 (PDF pp. 8–9): the printed ceiling mechanism has population contrast 1 - (16/625) times the independently mixed affected share. Restricted to this population contrast for a Binomial(4,2/5) baseline and independent affected status; omits the liar and false-confessor response rules, all rejection probabilities (including the approximately 80% and 30% examples), and all plotted finite-sample power curves and their dependence on sample size or violation share.", "headline": "Supplement Appendix C: the simulated design effect (“ceiling effect”) and the 80% power sentence; Figure A1, lower-left panel 'Violated assumption: no design effects'", "location": "Online supplement (author-appendix.pdf) p. 2, Appendix C, second paragraph and footnote 1; p. 3, first paragraph and Figure A1. Main-text counterpart: Section 4.1, last paragraph, pp. 50–51 (PDF pp. 8–9)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T16" }, { "declarations": [ { "claim_revision_id": "catalog:8c7ef874e32fd5ea83f645db49c32b4e631d26381ad7d91e673070da99dbb18e", "conditional_on": [], "credit_reason": "This is the positive population equality, with the strict remark handled by R131-08; finite observable support is without loss for a fixed bounded list. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.prevalence_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L523", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8c7ef874e32fd5ea83f645db49c32b4e631d26381ad7d91e673070da99dbb18e", "conditional_on": [], "credit_reason": "The finite representation loses no case of the bounded observable distribution. No monotonicity assumption is hidden in the Study structure. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.combined_excess", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L489", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:8c7ef874e32fd5ea83f645db49c32b4e631d26381ad7d91e673070da99dbb18e", "conditional_on": [], "credit_reason": "The printed inference E[X|Y=1]<1 fixes failure of monotonicity as positive-probability false confessions, rather than a change on a null atom. No claim-stage verdict changed after reading the record.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.false_confessions_strict", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L538", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:8c7ef874e32fd5ea83f645db49c32b4e631d26381ad7d91e673070da99dbb18e", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T17", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T17" }, "html": { "body": "

Lemma 1, pp. 47–48 (PDF pp. 5–6): under Assumptions 1–4 the combined functional identifies prevalence. Under Assumptions 2–4 its excess is the false-confession mass, equal to Pr(Y=1)(1-E[X|Y=1]); failure of almost-sure monotonicity is equivalent to positive false-confession mass and to E[X|Y=1] < 1, and gives strict overestimation. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.

", "headline": "Lemma 1, equation (1), and the remark after its proof", "location": "Lemma 1: p. 47 (PDF p. 5). Proof and remark: p. 48 (PDF p. 6)" }, "id": "P019-T17", "text": { "body": "Lemma 1, pp. 47–48 (PDF pp. 5–6): under Assumptions 1–4 the combined functional identifies prevalence. Under Assumptions 2–4 its excess is the false-confession mass, equal to Pr(Y=1)(1-E[X|Y=1]); failure of almost-sure monotonicity is equivalent to positive false-confession mass and to E[X|Y=1] < 1, and gives strict overestimation. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.", "headline": "Lemma 1, equation (1), and the remark after its proof", "location": "Lemma 1: p. 47 (PDF p. 5). Proof and remark: p. 48 (PDF p. 6)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T17" }, { "declarations": [ { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'GP019B2b.lean:671 states convergence of the actual sequence n·Var(μhat) under the iid sample law to sigmaSq, whose definition at P019_Audit.lean:1115 reproduces equation (3) in T18. Study.standing makes Assumptions 1–4 explicit, as maintained in the supplied standing model, and arbitrary fixed empty-cell means c1,c0 are universally quantified rather than imposed as favorable values. These completions fill undefined ratios without removing a defined paper case, and finite observable support preserves the bounded-list sampling distributions.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.proposition_one_variance_limit_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B2b.lean#L671", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The aggregate variance-limit proof already supplies the required estimator algebra.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.estimator_product_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1262", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The aggregate variance-limit proof already supplies the required estimator remainder argument.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.estimator_error_decomposition", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1270", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The proved variance-limit row already incorporates these supporting population influence moments.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.combined_influence_moments", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1160", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The proved variance-limit row already identifies the influence variance with equation (3).", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.combined_influence_variance", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1232", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The actual-variance limit is supplied directly by the aggregate row; this auxiliary sample calculation is not separately invoked.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.direct_sample_moments", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1840", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "conditional_on": [], "credit_reason": "The selected aggregate result already incorporates the identified nonconfessor contrast.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.nonconfessor_population_contrast", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1864", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:c4c3c1e4cedd599e19fb0a8938df03ee0b77cde0b6c37fada86b6f0607972217", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T18", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T18" }, "html": { "body": "

Proposition 1, p. 49 (PDF p. 7), Appendix B.1, p. 63 (PDF p. 21): under the maintained Assumptions 1–4, n times the actual variance of the combined estimator converges to equation (3), for every fixed real empty-cell completion. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; the theorem does not state the extension to arbitrary measurable underlying spaces with bounded list counts.

", "headline": "Proposition 1, equation (3)", "location": "p. 49 (PDF p. 7); proof in Appendix B.1, p. 63 (PDF p. 21)" }, "id": "P019-T18", "text": { "body": "Proposition 1, p. 49 (PDF p. 7), Appendix B.1, p. 63 (PDF p. 21): under the maintained Assumptions 1–4, n times the actual variance of the combined estimator converges to equation (3), for every fixed real empty-cell completion. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; the theorem does not state the extension to arbitrary measurable underlying spaces with bounded list counts.", "headline": "Proposition 1, equation (3)", "location": "p. 49 (PDF p. 7); proof in Appendix B.1, p. 63 (PDF p. 21)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T18" }, { "declarations": [ { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'GLP019B2.lean:774 exports root-n tightness, the centered root-n Gaussian limit with variance sigmaSq, convergence in probability of n·Varhat to sigmaSq, and Wald coverage tending to 1−α for every 0<α<1, matching T19. WeakLimit, ProbabilityLimit, and RootNTight at P019_Audit.lean:1545–1558 have their usual probabilistic meanings, while the estimators at lines 1071 and 1105 agree with the printed formulas wherever defined. Fixed arbitrary completions on deficient-count samples and the finite observable-law representation exclude no case of the stated asymptotic claim.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.combined_asymptotic_inference_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GLP019B2.lean#L774", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "conditional_on": [], "credit_reason": "The aggregate inference theorem already proves consistency of the stated sample plug-in variance.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.plugin_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1289", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "conditional_on": [], "credit_reason": "The aggregate inference theorem already incorporates this supporting population algebra.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.population_variance_term", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1295", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "conditional_on": [], "credit_reason": "This is the auxiliary positivity needed for the nondegenerate Gaussian and Wald limit, not a replacement for the inference theorem. No change from claim-stage.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.asymptotic_variance_positive", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1123", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "conditional_on": [], "credit_reason": "The selected aggregate result already incorporates the identified nonconfessor contrast.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.nonconfessor_population_contrast", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1864", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false } ], "formalization": { "claim_revision_id": "catalog:f181885f21e875bab9b568c66f7d82977c0aea0d57693cd458eb720b5e52249d", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T19", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T19" }, "html": { "body": "

text after Proposition 1 and Corollary 1, p. 49 (PDF p. 7): under Assumptions 1–4 the combined estimator is root-n tight and asymptotically normal, its plug-in variance is consistent, and its Wald intervals have limiting nominal coverage. Empty means and deficient-cell variance estimates use arbitrary fixed real completions. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; arbitrary measurable underlying spaces with bounded list counts are omitted.

", "headline": "Unnumbered statement after Proposition 1 (root-n consistency, asymptotic normality, the plug-in variance estimator) and Corollary 1", "location": "p. 49 (PDF p. 7)" }, "id": "P019-T19", "text": { "body": "text after Proposition 1 and Corollary 1, p. 49 (PDF p. 7): under Assumptions 1–4 the combined estimator is root-n tight and asymptotically normal, its plug-in variance is consistent, and its Wald intervals have limiting nominal coverage. Empty means and deficient-cell variance estimates use arbitrary fixed real completions. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; arbitrary measurable underlying spaces with bounded list counts are omitted.", "headline": "Unnumbered statement after Proposition 1 (root-n consistency, asymptotic normality, the plug-in variance estimator) and Corollary 1", "location": "p. 49 (PDF p. 7)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T19" }, { "declarations": [ { "claim_revision_id": "catalog:66dd45b9455c77890af5c7497df7800759a4d6018ff3cf5350520bf3a7634d3b", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'GP019B2b.lean:779 exports both actual scaled variance limits and the strict inequality sigmaSqProposition 2, p. 49 (PDF p. 7): under Assumptions 1–4, both scaled actual variance limits exist and the combined estimator's limit is strictly smaller than the standard estimator's, for all fixed real empty-cell and empty-arm completions. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; the two variance limits and strict comparison are not stated for arbitrary measurable underlying spaces with bounded list counts.

", "headline": "Proposition 2", "location": "p. 49 (PDF p. 7); proof in Appendix B.1, pp. 64–65 (PDF pp. 22–23), ending at display (10)" }, "id": "P019-T20", "text": { "body": "Proposition 2, p. 49 (PDF p. 7): under Assumptions 1–4, both scaled actual variance limits exist and the combined estimator's limit is strictly smaller than the standard estimator's, for all fixed real empty-cell and empty-arm completions. Restricted to iid draws on a finite underlying probability space with arbitrary real atom masses; the two variance limits and strict comparison are not stated for arbitrary measurable underlying spaces with bounded list counts.", "headline": "Proposition 2", "location": "p. 49 (PDF p. 7); proof in Appendix B.1, pp. 64–65 (PDF pp. 22–23), ending at display (10)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T20" }, { "declarations": [ { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/Wave1/Gap/GP019B3b.lean:367 states beta = 1 under all four printed assumptions, the standard-normal studentized limit, and every interior CDF limit for the displayed two-sided p-value, matching Proposition 3. The finite observable law in P019/P019_Audit.lean:24 loses no cases because the fixed list counts and binary variables have finite joint range and the atom masses are arbitrary real probabilities. The explicit positive cell probabilities in P019/P019_Audit.lean:102 follow from treatment independence and positive variances of binary Y and Z. The completions in P019/P019_Audit.lean:1060, P019/P019_Audit.lean:1603, and P019/P019_Audit.lean:1613 change only undefined finite-sample expressions; positive cell probabilities and positive limiting variance make these exceptional events asymptotically negligible for every fixed completion. Thus the noted conventions do not narrow the claimed large-sample result or make it vacuous.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.proposition_three_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B3b.lean#L366", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "This is an acceptable generalization of the printed beta=1 assertion and a proof of its weaker-assumption correction; it is not credited as the full sample-inference theorem. No change from claim-stage.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_one_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L425", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "The aggregate placebo-I theorem already includes the completed ratio-statistic argument.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.ratio_residual_identity", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1279", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "The aggregate placebo-I theorem already supplies the full inferential conclusion using the response residual calculation.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_influence_residual", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L1884", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "The aggregate placebo-I theorem already supplies the full inferential conclusion using its supporting moments.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_influence_moments", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L2187", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "conditional_on": [], "credit_reason": "The aggregate placebo inference theorem already exports every interior p-value CDF limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.normal_pvalue_uniform_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B1.lean#L116", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:a7f4aabbfc086e59cb8621a09e5ed800da742bea572d2bd679167606eb383d19", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T21", "decision": "whole", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T21" }, "html": { "body": "

Section 4.1 and Proposition 3, p. 50 (PDF p. 8): Assumptions 1–3 and defined confessor cells imply equality of the control conditional law and the treated law shifted down by one. Consequently every level-alpha test of that distributional null remains level-alpha under these assumptions. Under Assumptions 1–4, beta = 1, the studentized estimator tends to a standard normal, and the printed two-sided p-value is asymptotically uniform, for every fixed completion. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.

", "headline": "Proposition 3 (Placebo Test I)", "location": "p. 50 (PDF p. 8)" }, "id": "P019-T21", "text": { "body": "Section 4.1 and Proposition 3, p. 50 (PDF p. 8): Assumptions 1–3 and defined confessor cells imply equality of the control conditional law and the treated law shifted down by one. Consequently every level-alpha test of that distributional null remains level-alpha under these assumptions. Under Assumptions 1–4, beta = 1, the studentized estimator tends to a standard normal, and the printed two-sided p-value is asymptotically uniform, for every fixed completion. Usual measure-theoretic conditions are assumed; the underlying probability space is arbitrary.", "headline": "Proposition 3 (Placebo Test I)", "location": "p. 50 (PDF p. 8)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T21" }, { "declarations": [ { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "conditional_on": [], "credit_reason": "{'source': 'adjudication', 'decision': 'rejected', 'what': 'P019/Wave1/Gap/GP019B3b.lean:399 states delta = 0, the standard-normal studentized limit, and asymptotic uniformity of the printed p-value under treatment independence and Assumption 4, matching the inferential component of Proposition 4. Assumption 4 is expressly required by that component and included in the supplied standing assumptions, so bundling the population identity with inference imposes no additional restriction on this component; the standalone identity under treatment independence and defined treatment arms is separately stated at P019/P019_Audit.lean:548. The finite law preserves every observable distribution allowed by the fixed finite lists, and the positive cell probabilities in P019/P019_Audit.lean:102 are implied by independence and nondegenerate binary Y and Z. The completed means and variance estimator at P019/P019_Audit.lean:1631 and P019/P019_Audit.lean:1648 agree with the printed formulas whenever defined, while the totalized p-value at P019/P019_Audit.lean:1613 only supplies an exceptional finite-sample convention. Positive arm probabilities and positive conditional Y variance make all these conventions irrelevant to the claimed limits, so the minor notes identify no material defect.'}", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.proposition_four_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B3b.lean#L396", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "conditional_on": [], "credit_reason": "Positive arm probabilities make the printed conditional expectations defined. IID sampling is unnecessary for this population implication, and no monotonicity, list equation, or inferential conclusion is asserted.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.placebo_two_identity", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L548", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true }, { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "conditional_on": [], "credit_reason": "The auxiliary converse for Y-Z independence is not a clause of the retained Proposition 4 inference assertion.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.binary_null_characterization", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L2095", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "conditional_on": [], "credit_reason": "The aggregate placebo-II theorem already proves the inferential bundle; this row alone contains only single-subject influence moments.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.direct_influence_moments", "manifest_use": "not_used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/P019_Audit.lean#L2250", "status": "proved", "support_scope": "historical declaration", "used_in_solution": false }, { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "conditional_on": [], "credit_reason": "The aggregate placebo inference theorem already exports every interior p-value CDF limit.", "credit_status": "ok", "fidelity_status_baseline": "ok", "lean_name": "AronowAudit.P019.Wave1.normal_pvalue_uniform_proof", "manifest_use": "used", "named_axioms": [], "says": null, "source": "https://github.com/pmaronow/llm-audit-lean/blob/b9e9df44831afed4f2b8f63be5c0cc9169a5fda4/AronowAudit/P019/Wave1/Gap/GP019B1.lean#L116", "status": "proved", "support_scope": "historical declaration", "used_in_solution": true } ], "formalization": { "claim_revision_id": "catalog:5aa4bd7b8ce9dcada41ca2ad1236127dcd559bd3fce1bc29e9cfe0f30e97c563", "crosswalk_url": "../../data/lean-crosswalk.json#/targets/P019-T22", "decision": "restricted", "release_id": "sap-2026-11-08-assured-release", "target_id": "P019-T22" }, "html": { "body": "

Proposition 4, p. 51 (PDF p. 9): Assumption 3 and positive treatment arms give delta = 0; with Assumption 4, the studentized estimator has a standard normal limit and the printed two-sided p-value is asymptotically uniform for every fixed completion. Restricted to finite underlying probability spaces with arbitrary real atom masses; the extension to arbitrary measurable spaces, the general randomized causal-effect interpretation, and the positive delayed-treatment design implication are not formalized. The additional finite protocol shows failure of independence when assignment is disclosed before the direct answer and only physical list display is delayed. This does not refute the printed design assertion. Usual measure-theoretic conditions are automatic for these finite models.

", "headline": "Proposition 4 (Placebo Test II)", "location": "p. 51 (PDF p. 9)" }, "id": "P019-T22", "text": { "body": "Proposition 4, p. 51 (PDF p. 9): Assumption 3 and positive treatment arms give delta = 0; with Assumption 4, the studentized estimator has a standard normal limit and the printed two-sided p-value is asymptotically uniform for every fixed completion. Restricted to finite underlying probability spaces with arbitrary real atom masses; the extension to arbitrary measurable spaces, the general randomized causal-effect interpretation, and the positive delayed-treatment design implication are not formalized. The additional finite protocol shows failure of independence when assignment is disclosed before the direct answer and only physical list display is delayed. This does not refute the printed design assertion. Usual measure-theoretic conditions are automatic for these finite models.", "headline": "Proposition 4 (Placebo Test II)", "location": "p. 51 (PDF p. 9)" }, "url": "https://pmaronow.github.io/llm-audit/works/P019/#P019-T22" } ] }, "missing_sources": [], "reproduction": { "agreement": "agrees_in_part", "status": "partial", "targets": [ { "agreement": "to_rounding", "location": "p. 54–55 (PDF pp. 12–13)", "method": "authors' code with changes", "status": "reproduced", "target": "Table 3 and Figure 1" }, { "agreement": "to_rounding", "location": "p. 56–57 (PDF pp. 14–15)", "method": "authors' code with changes", "status": "reproduced", "target": "Table 6 and Figure 2" }, { "agreement": "to_rounding", "location": "p. 55–56 (PDF pp. 13–14)", "method": "authors' code with changes", "status": "reproduced", "target": "Tables 4–5" }, { "agreement": "to_rounding", "location": "p. 57–58 (PDF pp. 15–16)", "method": "authors' code with changes", "status": "reproduced", "target": "Tables 7–8" }, { "agreement": "to_rounding", "location": "Online supplement pp. 6–7 (PDF pp. 6–7); main p. 58 (PDF p. 16)", "method": "authors' code with changes", "status": "reproduced", "target": "Supplement Tables A2 and A5" }, { "agreement": "to_rounding", "location": "Online supplement pp. 5–7; main pp. 58–59 (PDF pp. 16–17)", "method": "authors' code with changes", "status": "reproduced", "target": "Supplement Tables A3–A4 and A6–A7; replication placebo claims" }, { "agreement": "to_rounding", "location": "Online supplement pp. 4–5 (PDF pp. 4–5)", "method": "authors' code with changes", "status": "reproduced", "target": "Supplement Table A1 and question-order effects" }, { "agreement": "agrees_in_part", "location": "Main pp. 50–51 (PDF pp. 8–9); online supplement pp. 2–3 (PDF pp. 2–3)", "method": "I reran eight selected power anchors with changes to the authors' code and compared the described and coded design effects. False-confession and list-lying anchors agree approximately; at 800 admitters and 20% design-affected respondents the described rule gives about 5% power and the code about 78%, against the reported 80%. The full grid was not recomputed.", "status": "recalculated", "target": "Figure A1 and Placebo I power anchors" } ] }, "summary": "The paper proposes an estimator of the prevalence of a sensitive behavior that combines a list experiment with a direct question. It shows that, under its assumptions, the estimator is asymptotically more efficient than the standard list estimator, derives two placebo tests of the assumptions, and applies them in survey experiments. 10 entries are classified as errors (1 Moderate, 9 Minor), and 4 are clarity notes. The main finding is \\fid{P019-E004}: the supplement's power panel for design effects simulates a different design effect from the one described.", "what_was_checked": "I checked the following, with the qualifications stated.\n\\begin{itemize}\n\\item Section 2 and Lemma 1: the probabilities $\\Pr[Y_i=0]=1-\\mu+\\mu(1-p)$, $\\Pr[Y_i=1]=\\mu p$ and $\\Pr[X_i=1\\mid Y_i=0]=(1-p)\\mu/(1-\\mu p)$, the identity (1), and the remark that the expression exceeds $\\mu$ without Monotonicity. I derived the identity again, and it holds to rounding error on 20,000 discrete models, drawn at random, that satisfy Assumptions 1--3. With false confessions the excess is $\\Pr[X_i=0,Y_i=1]$. The slips in the proof are in \\fid{P019-E008}.\n\\item Proposition 1 and Corollary 1: formula (3) holds under Assumptions 1--4, by a derivation from the influence function that does not use the proof in Appendix B. It agrees with 243 cases computed in exact arithmetic. In a simulation with $n=1000$, $n$ times the variance is 4.10 against 4.08 from (3), and the 95\\% interval of Corollary 1 covers $\\mu$ in 94.95\\% of samples. The variance estimator printed on p.~49 is the plug-in of the delta-method variance, and the replication code implements it as printed. The hypothesis of Proposition 1 is the subject of \\fid{P019-E002}, and the proofs in Appendix B are the subject of \\fid{P019-E003}.\n\\item Proposition 2 and each algebraic step of its proof: the decomposition of the variances, the substitution that uses $\\beta=1$, the first-order condition, the minimized value, and strictness from Assumption 4. The efficiency gap is a strictly positive term plus a nonnegative square.\n\\item Propositions 3 and 4: $\\beta=1$ and $\\delta=0$ under their nulls, the variance estimators, and the two printed formulas for the p-value. The wording around them is the subject of \\fid{P019-E001}, \\fid{P019-N001} and \\fid{P019-N006}.\n\\item The tables: all 548 printed numeric entries of the tables in the main text and the supplement are reproduced at the printed precision from the two data files in the replication files. These include the reductions in sampling variance (14.0\\% to 67.1\\% in Study A, 16.4\\% to 66.0\\% in Study B) and the sample sizes ($1{,}023-9=1{,}014$, with 500 in Study A and 514 in Study B, and $1{,}019-7=1{,}012$ in the replication). Placebo Test I rejects in two of five cases in Study A and in none in Study B, and in one and three of five in the replication, six of twenty in all. The labels of some tables are corrected in \\fid{P019-E009}.\n\\item Section 5.3 and Appendix E, for the original experiment: no difference between Studies A and B is significant at 0.05 for the direct estimates (largest $|z|$ 1.74) or for the combined estimates (largest $|z|$ 1.00). Two differences are significant for the standard list estimates (nuclear power and CNN, with $z=2.04$ and $-2.11$). Fisher's method for Placebo Test I in Study B gives the statistic $12.14$ on 10 degrees of freedom and $p=0.276$, which is insignificant, as the text says. Its reference distribution is the subject of \\fid{P019-N003}.\n\\item Section 5.4: the ten differences between the original experiment and the replication in the standard list estimates are all nonsignificant (smallest $p$ about 0.151). The effect of the list treatment on the direct answer about CNN is $13.2$ percentage points ($p=0.0025$) in Study B and $9.2$ points ($p=0.038$) in Study B of the replication.\n\\item Figures 1 and 2: the plotted estimates and the intervals of $1.96$ standard errors agree with Tables 3 and 6, and the combined intervals are the narrower ones for all ten topics. The intervals cut off at zero are noted in \\fid{P019-E010}.\n\\item Power of Placebo Test I for false confessors: 0.285 when 20\\% of 200 admitters confess falsely and 0.789 when 20\\% of 800 do, against ``about 30'' and ``approximately 80 percent'' in Section 4.1. The power is 0.800 when 20\\% of 800 lie on the list. Each value is from 4,000 simulations. The grid described in Appendix C agrees with the code. The panel for design effects is the subject of \\fid{P019-E004}, the deposited scripts are the subject of \\fid{P019-E005}, and the first panel of Figure A2 is the subject of \\fid{P019-E006}.\n\\end{itemize}\nMy checks had the following limits. The works that the paper cites were not consulted, and its statements about other work were not checked against the sources. The full grid of Figure A1, with 7,676 cells of 1,000 simulated experiments each, was not computed again; only the points quoted in the text and a few others were. The deposited scripts set no seed for that figure. Section 5.4 says: ``The list experimental estimates vary somewhat between the original experiments and the replication, but none of these differences are significant ($p>0.05$).'' I did not confirm an error in this sentence: it holds for the standard list estimates, while one of ten combined-estimate differences has $p$ about 0.021. The broader reading is not established. The reported hourly rate of \\$15.45 could not be checked, since neither the article, the supplement nor the deposited data reports completion times. Parts of the later software package that do not concern this paper were not read. No separate correction or erratum was available to me, and I make no claim that none exists. That this audit lists no further corrections is not a finding that the rest of the paper is correct.", "work": { "arxiv_id": "1312.1268", "audit_complete": true, "authors": [ "P. M. Aronow", "Alexander Coppock", "Forrest W. Crawford", "Donald P. Green" ], "category": "Articles - Theory and Methodology", "citation": "P. M. Aronow, Alexander Coppock, Forrest W. Crawford, and Donald P. Green (2015). Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence. Journal of Survey Statistics and Methodology 3:43–66.", "doi": "10.1093/jssam/smu023", "edition": { "pages_in_audited_file": 24, "sha256": "88f6543a1abdfd63a18e6c27268bf85bbb58aaa75464a6f63d5e1b443223ec53", "source_url": "https://doi.org/10.1093/jssam/smu023", "venue": "Journal of Survey Statistics and Methodology", "year": "2015" }, "id": "P019", "later_materials": null, "title": "Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence", "type": "Journal article", "url": "https://pmaronow.github.io/llm-audit/works/P019/", "venue": "Journal of Survey Statistics and Methodology", "year": "2015" } }