Letter P054

Corrections to: Policy Misperceptions and Support for Gun Control Legislation

P. M. Aronow and Benjamin T. Miller (2016). Policy Misperceptions and Support for Gun Control Legislation. The Lancet 387(10015):223.

DOI: 10.1016/S0140-6736(16)00042-8
Audited edition: The Lancet, 2016, 387(10015):223; 1 page in the audited file
SHA-256 of the audited file: cf8325e4f625073599232e6abbbfd26b98a0d4b961458facead1f12c0ac0a08e
Permanent address: https://pmaronow.github.io/llm-audit/works/P054/

Summary

This one-page letter asks whether mistaken beliefs about existing gun laws contribute to the lack of public mobilisation for new legislation. In a survey from the 2014 Cooperative Congressional Election Study ( $n = 1384$ ), 77% favoured universal background checks, 53% preferred stricter gun laws, and 41% believed that universal checks are already federal law. 4 entries are classified as errors (1 Moderate, 3 Minor); 4 other concerns were not confirmed. Under an independent-respondent survey approximation with the released weights, all nine 95% intervals widen and the stricter-law interval includes 50%; the nine estimates and three main positive comparisons remain (P054-N001). The letter reports no majority hypothesis test, and coverage under the full survey design is not established. I report using intercept-only weighted least squares in R with classical standard errors; independent fits reproduce the rounded numbers but do not identify the original procedure.

What was and was not checked

The letter is one page, with five references and no table, figure, formula or appendix, and I read all of it. I checked the following.

Every number quoted here from the letter was confirmed on the page.
The letter’s descriptions of the survey questions agree with the questionnaire and the codebooks of the survey. The questions are these: support for background checks for all gun buyers; the knowledge question with its three answers (all purchases, some, none); whether one could still vote for someone standing for office with whom one agrees on other issues but disagrees on gun control; and whether gun laws should be more strict, kept as they are, or less strict.
The three knowledge percentages, 41, 47 and 12, sum to 100. The three comparisons imply differences of 24, 15 and 37 percentage points, and each printed interval contains its estimate.
The codebooks record 1393 post-election respondents, and $n = 1384$ is attainable as the number with valid answers to the first three questions about guns ( $1393 - 1 - 4 - 4$ ).
The printed 77% and 12% agree with the unweighted codebook counts, $1066/1389 = 76.7\%$ and $172/1389 = 12.4\%$ . The printed 53%, 41% and 47% cannot be unweighted proportions on 1384 of these respondents, and they are compatible with weighted ones.
Each of the nine printed intervals agrees at the printed precision with $p \pm 1.96\sqrt{p(1-p)/n}$ for a suitable proportion and group size. The apparent asymmetry of 53 (51–56) and 41 (38–43) can arise from rounding. This is a check of internal consistency and not of the method behind the intervals.
A recalculation from the respondent-level file of the February 2015 release, with the team weight, on the 1384 respondents with valid answers to the first three questions about guns, reproduced to rounding all nine printed percentages. With $p \pm 1.96\sqrt{p(1-p)/n}$ , where $n$ is the unweighted size of the group, it also reproduced all nine printed intervals. These choices give a compatible reconstruction; agreement does not uniquely identify the original release, weight, sample rule or interval method, which the letter does not document.
Under an independent-respondent survey linearisation with the released weights, all nine intervals widen; retaining covariance, the three reported differences retain 95% intervals that exclude zero (P054-N001). The letter itself does not specify the interval method.
Among respondents who did not strongly favour universal background checks, the same weighted calculation gives 30% support for stricter laws among those who believed checks cover all purchases and 16% among those who knew they cover only some. This opposite direction supports the letter’s “no such relation” in its natural meaning of no such positive relation.
The engagement comparison’s 71% and 34% are reproduced without restricting the groups to voters. The reconstructed groups include 18 and 149 respondents who explicitly reported not voting, so “voters” should be replaced by “respondents” (P054-N005).
For references 2 and 3, the titles, journals, years, volumes and pages agree with the publishers' pages. The author names are corrected in P054-N006 and P054-N007.

The following were not checked. References 1, 4 and 5 were not compared with their sources, and neither were the statements that rest on them: more than 30 000 firearm deaths a year, and 92% of 1446 people polled. Which sales of guns are exempt from background checks under federal law was not researched. The suggestion that public education could mobilise support is framed as a possibility. The comparisons in the letter contain no information treatment and cannot identify an effect of education.

No analysis code for the letter was identified, and the letter does not say which release of the survey data or which weights it used. No supplement or published correction was identified. The absence of further corrections is not a finding that the letter is correct.

I report that the original analysis used intercept-only weighted least squares in R with classical standard errors. Actual fits reproduce all nine survey percentages and their interval endpoints after rounding, establishing compatibility without independently identifying the original procedure. Under this survey linearisation the three principal difference intervals remain positive. For stricter-law support, the interval reconstructed with $p \pm 1.96\sqrt{p(1-p)/n}$ is $50.60848$ – $55.86594\%$ , while the survey-linearised interval is $49.51388$ – $56.96055\%$ , which includes $50\%$ . The paper does not explicitly report a majority hypothesis test, and these calculations do not establish coverage under the full national survey design or justify the weighting and response assumptions needed for population inference.

Headline claims and related corrections

The letter reports a nationally representative survey fielded through the 2014 Cooperative Congressional Election Study, with $n=1384$ .
No numbered result (the letter has no tables, figures or numbered results). In-text statement of the sample, p. 223 (PDF p. 1), column 1, second paragraph; codebook counts for YAL401-YAL405.
See P054-N001.
77% (95% CI 75-79) of Americans favoured universal background checks, yet only 53% (51-56) reported preferring stricter gun control laws: support for universal checks is substantially greater than preference for stricter laws.
No numbered result (the letter has no tables, figures or numbered results). In-text estimates, p. 223 (PDF p. 1), column 1, second paragraph; survey items YAL402 and YAL401 (or YAL405).
See P054-N001.
41% (38-43) of Americans believed universal background checks are already required by federal law, 47% (44-50) correctly said a check is required only for some purchases, and 12% (10-14) believed no federal check is required.
No numbered result (the letter has no tables, figures or numbered results). In-text estimates, p. 223 (PDF p. 1), column 1, second paragraph; survey item YAL403.
See P054-N001.
Among people who strongly supported universal background checks, correct knowledge of the law predicted support for stricter gun laws: 74% (69-79) of those who wrongly believed checks are universal supported stricter laws against 89% (86-92) of those who knew some sales are exempt. No such relation was recorded among those who did not strongly favour universal background checks.
No numbered result (the letter has no tables, figures or numbered results). In-text estimates, p. 223 (PDF p. 1), end of column 1 and top of column 2; cross-tabulation of YAL402, YAL403 and YAL401 (or YAL405).
See P054-N001.
There is an engagement gap: among voters who thought gun laws should be less strict, 71% (65-78) said they would never vote for a candidate who did not share their position on gun control, compared with 34% (30-37) of those who support stricter gun laws.
No numbered result (the letter has no tables, figures or numbered results). In-text estimates, p. 223 (PDF p. 1), column 2, second paragraph; survey items YAL404 and YAL401.
See P054-N001, P054-N005.
The letter notes that many people mistakenly believe strong gun laws are already enacted and do not support new legislation. It suggests that poor awareness of existing law might contribute to the difference between 77% support for universal checks and 53% preference for stricter laws, and that public education could help mobilise support.
No numbered result (the letter has no tables, figures or numbered results). Interpretation of the in-text $77\%$ versus $53\%$ contrast, the $41\%$ misperception estimate and the $74\%$ (interval $[69\%,79\%]$ ) versus $89\%$ (interval $[86\%,92\%]$ ) comparison among strong supporters; p. 223 (PDF p. 1), column 1, second paragraph, and the closing sentences of column 2. No education intervention is reported.

Corrections

Moderate (1)

A real substantive error with limited reach.

P054-N001 Moderate

Under the stated independent-respondent survey linearisation, all nine intervals widen and the 53% stricter-law interval includes 50%; the nine estimates and the three main positive comparisons remain.

Columns 1 and 2, the nine percentages printed with 95% confidence intervals, from “77% (95% CI 75–79)” to “34% (30–37)”, p. 223 (PDF p. 1)

See also under Formal statements (Lean): P054-T1, P054-T2

I disputed this classification, but as a rule, deferred to the models’ final judgment: see my comment and the history of this entry.

As printed

To answer this question, we undertook a nationally representative survey through the 2014 Cooperative Congressional Election Study (n=1384). […] 77% (95% CI 75–79) of Americans favoured universal background checks, yet only 53% (51–56) reported preferring stricter gun control laws. […] We found that 41% (38–43) of Americans believed that universal background checks are already required by federal law, whereas 47% (44–50) correctly reported that a federal background check is required only for some gun purchases, and 12% (10–14) believed that no federal background checks are required.

[…] Whereas 74% (95% CI 69–79) of those who incorrectly believed that the USA has universal background checks supported stricter gun laws, 89% (86–92) of those who knew that some gun sales do not require background checks supported stricter laws. […] 71% (95% CI 65–78) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34% (30–37) of those who support stricter gun laws.

The problem

The letter states neither its weights, nor the rule that gives $n=1384$ , nor how its intervals were computed. The public survey files reproduce all nine printed percentages and intervals under the reconstruction below. With the team weight of the February 2015 release, on the 1384 respondents with valid answers to the first three questions about guns, each of the nine percentages agrees with the print after rounding. Let $\hat p$ be one of these weighted proportions and let $n$ be the unweighted number of respondents in its group. Then the interval $\hat p\pm1.96\sqrt{\hat p(1-\hat p)/n}$ rounds to the printed interval in all nine cases, at both endpoints.

This interval uses the weights in $\hat p$ but does not account for their unequal influence in the variance. It treats $\hat p$ as the proportion in a simple random sample of $n$ respondents with equal weights. The weights of the 1384 respondents are unequal: they range from $0.05$ to $7.00$ . The usual linearised variance of a weighted proportion, $V$ in the display under Evidence, allows for them. With $V$ , every standard error is larger, by a factor ranging from approximately $1.28$ to $1.47$ , and every interval is wider. For example, where the letter prints 74% (69–79), the interval becomes 74% (67–81). Under this approximation, the printed intervals overstate the precision of all nine estimates. No percentage changes.

Two limits apply. First, the rounded figures do not identify the original calculation uniquely. An actual intercept-only weighted least-squares fit in R, with classical standard errors and either normal or default $t$ critical values, also gives the same nine intervals after rounding, and six other rules for selecting respondents reproduce every printed percentage and interval. I report using classical intercept-only weighted least squares in R for the original analysis. That method is compatible with every printed number, while its historical use is not independently established by the letter, the available deposit or rounded agreement. Each matching calculation omits the squared-weight residual term from the survey-ratio variance, and under each of the seven rules none of the nine intervals based on the same linearisation rounds to the printed one. Second, $V$ is the ordinary approximation for independent respondents. Each release of the public files holds at most one weight and no strata, clusters or replicate weights, and the files do not document how the weight was built. The variance under the full design of the survey cannot be computed from them, and whether any of these intervals covers the national proportion 95% of the time is not established.

Evidence a recalculation from the source

I recalculated the nine estimates from the public survey files (2014 Cooperative Congressional Election Study, Yale team module, Harvard Dataverse, https://doi.org/10.7910/DVN/FBXY8V). Exact rational comparisons confirmed the rounded percentages and endpoints of both intervals in the table.

The following notation is introduced for the recalculation; the letter prints no formula. Fix one of the nine estimates. For respondent $i$ , let $d_i\in\{0,1\}$ indicate that $i$ belongs to the group and gave a valid answer, let $y_i\in\{0,1\}$ be the answer, and let $w_i>0$ be the weight. The sums below run over the $N=1384$ respondents. Set $n=\sum_i d_i$ , $W=\sum_i d_iw_i$ and $\hat p=\sum_i d_iw_iy_i/W$ , and define

V_0=\frac{\hat p(1-\hat p)}{n},\qquad V=\frac{N}{N-1}\cdot\frac{\sum_i d_iw_i^2(y_i-\hat p)^2}{W^2}.

The variance $V$ comes from linearising the ratio $\hat p=\bar a/\bar b$ , where $\bar a=N^{-1}\sum_i d_iw_iy_i$ and $\bar b=W/N$ . To first order, a change in $(\bar a,\bar b)$ changes the ratio by $(\mathrm{d}\bar a-\hat p\,\mathrm{d}\bar b)/\bar b$ , so respondent $i$ enters through the residual $z_i=d_iw_i(y_i-\hat p)$ . These residuals sum to zero. Their empirical variance is $\sum_i z_i^2/(N-1)$ , and the estimated variance of their mean is $\sum_i z_i^2/(N(N-1))$ . Dividing by $\bar b^2=W^2/N^2$ gives $V$ . If all weights are equal, then $\sum_i d_iw_i^2(y_i-\hat p)^2/W^2=\hat p(1-\hat p)/n$ , and $V$ equals $V_0$ up to the factor $N/(N-1)$ . The standard errors $\sqrt V$ agree with those of the survey package of R for a design with independent respondents and these weights.

For an intercept-only weighted least-squares fit with classical standard errors, binary responses give

\sum_i d_iw_i(y_i-\hat p)^2=W\hat p(1-\hat p),\qquad V_{\mathrm{classical}}=\frac{\hat p(1-\hat p)}{n-1}.

Thus entering the team weights in R’s lm leaves a classical intercept variance differing from $V_0$ only by the factor $n/(n-1)$ . Numerical agreement with the print establishes compatibility with the method I report using, not its historical use or the inverse-variance assumptions that would justify its classical SEs. Both matching formulas remain distinct from the survey-ratio variance $V$ .

The estimates are coded as follows. Favouring universal checks is the answer “strongly favor” or “favor”, and strong support is “strongly favor”. Preferring stricter laws is the answer “more strict” to the question with three answers. The last two estimates are the share answering “No” to the question whether one could still vote for someone standing for office with whom one disagrees on gun control. Respondents without a valid answer to it are left out.

The nine estimates in the order of the letter. Printed: percentage and 95% interval as printed. $n$ : respondents in the group. $\hat p$ : weighted percentage. Fourth and fifth columns: the two intervals, in per cent. Rounded: the fifth column at the precision of the letter. Last column: ratio of the two standard errors.
Printed	$n$	$\hat p$ (%)	$\hat p\pm1.96\sqrt{V_0}$	$\hat p\pm1.96\sqrt{V}$	Rounded	$\sqrt{V/V_0}$
77 (75–79)	1384	76.82	74.60–79.05	73.68–79.97	74–80	1.41
53 (51–56)	1384	53.24	50.61–55.87	49.51–56.96	50–57	1.42
41 (38–43)	1384	40.82	38.23–43.41	37.08–44.56	37–45	1.44
47 (44–50)	1384	47.05	44.42–49.68	43.34–50.76	43–51	1.41
12 (10–14)	1384	12.13	10.41–13.85	9.67–14.58	10–15	1.43
74 (69–79)	306	73.90	68.98–78.82	66.86–80.95	67–81	1.43
89 (86–92)	334	89.15	85.81–92.48	84.89–93.40	85–93	1.28
71 (65–78)	191	71.16	64.74–77.59	62.52–79.81	63–80	1.35
34 (30–37)	746	33.82	30.43–37.22	28.85–38.80	29–39	1.47

In every row the fourth column rounds to the printed interval and the fifth does not. The ratio $V/V_0$ ranges from approximately $1.626$ to $2.147$ . It differs from one estimate to the next, since it depends on the answers as well as on the weights. No common rescaling factor was used. Without weights, or with the weights of the August 2015 release, the nine percentages are not all reproduced: the rounded values are 77, 54, 39, 49, 12, 76, 88, 71, 33 and 75, 53, 40, 48, 12, 73, 88, 73, 33. The five tested interval methods for a proportion in the survey package of R, applied to the same design, reproduce none of the nine printed intervals. If each group is treated as a separate sample, the factor $N/(N-1)$ becomes $n/(n-1)$ ; eight rounded intervals are unchanged, and the eighth row becomes 62–80.

What survives

All nine percentages are reproduced and stand, and the three knowledge percentages still sum to 100. The three comparisons that the letter draws keep 95% intervals that exclude zero when the same linearisation is applied to each difference. The difference between 77% and 53% is $23.59$ percentage points ( $19.83$ to $27.34$ ). The difference between 89% and 74% among strong supporters is $15.24$ points ( $7.01$ to $23.47$ ), and the difference between 71% and 34% is $37.34$ points ( $27.36$ to $47.32$ ). The printed precision of each estimate is lost under this approximation. One confidence inference also changes: the matching stricter-law interval is $50.60848$ – $55.86594\%$ , while the interval using $V$ is $49.51388$ – $56.96055\%$ . The latter contains $50\%$ , so a simple majority preferring stricter laws is no longer established by the reported two-sided 95% interval criterion. The paper does not explicitly report a majority hypothesis test, and this change does not reverse the $77\%$ -versus- $53\%$ comparison. Neither calculation establishes a causal education effect or statistical equivalence to zero among non-strong supporters. The group to which the last two estimates refer is corrected in P054-N005.

Corrected

77% (95% CI 74–80) of Americans favoured universal background checks, yet only 53% (50–57) reported preferring stricter gun control laws. […] An estimated 41% (37–45) of Americans believed that universal background checks are already required by federal law, whereas 47% (43–51) correctly reported that a federal background check is required only for some gun purchases, and 12% (10–15) believed that no federal background checks are required.

[…] Among those who strongly supported universal background checks, 74% (95% CI 67–81) of those who incorrectly believed that the USA has universal background checks supported stricter gun laws, whereas 89% (85–93) of those who knew that some gun sales do not require background checks supported stricter laws. […] Among respondents who thought gun laws should be less strict than at present, 71% (95% CI 63–80) said they would never vote for a political candidate who did not share their position on gun control, compared with just 34% (29–39) of those who support stricter gun laws.

The letter should also state the release of the data, the respondents selected, the weights and the interval method that were used. If they are those of the recalculation, a sentence such as the following would serve: Estimates use the team weight of the February 2015 release of the survey data, on the 1384 respondents with valid answers to the three questions on gun laws, on universal background checks and on existing federal law. The 95% confidence intervals use a linearised variance that allows for the weights and treats respondents as independent.

These replacement intervals are conditional on the following approximation; they are not independently established full-design national intervals. The corrected intervals are $\hat p\pm1.96\sqrt V$ , with $\hat p$ and $V$ as defined under Evidence. They are approximate 95% intervals under three conditions: respondents can be treated as independent draws with the released weights; the linear and normal approximations are adequate; and the released weights identify the national population, with the retained respondents, including those answering the engagement question, identifying the same proportions in each group. If the variance under the full design of the survey is available, it should replace $V$ . The printed limits could be kept only with a documented inferential justification that accounts for the survey design and response selection.

Crop of the second paragraph of the first column of page 223, 23 lines, from 'To answer this question, we undertook a nationally representative survey through the 2014 Cooperative Congressional Election Study (n=1384)' to '12 per cent (10-14) believed that no federal background checks are required.' Red outlines surround five ranges: 75-79, which follows '77 per cent (95 per cent CI', and the ranges in parentheses (51-56), (38-43), (44-50) and (10-14), which follow 53, 41, 47 and 12 per cent. — Column 1, second paragraph, p. 223 (PDF p. 1). The outlines mark the first five printed intervals: 75–79, 51–56, 38–43, 44–50 and 10–14.

Crop of the first 22 lines of the second column of page 223, from 'existing laws predicted support for stricter gun laws' to 'laws. However, the role of policy'. Red outlines surround four ranges: 69-79, which follows '74 per cent (95 per cent CI'; (86-92), which follows 89 per cent; 65-78, which follows '71 per cent (95 per cent CI'; and (30-37), which follows 34 per cent. Between them the text reads 'We recorded no such relation among those who did not strongly favour universal background checks' and 'We also identified evidence of an engagement gap: among voters who thought gun laws should be less strict than at present'. — Column 2, first 22 lines, p. 223 (PDF p. 1). The outlines mark the other four printed intervals: 69–79, 86–92, 65–78 and 30–37.

Chart titled 'Allowing for the survey weights widens all nine 95 per cent intervals', with the subtitle 'Standard errors are 1.28 to 1.47 times larger; the nine percentages do not change'. The horizontal axis is percentage points below or above the estimate, from -9 to +9. Each of nine rows has an orange interval, which rounds to the printed one, above a wider blue interval, which allows for the survey weights; both are centred at zero. To the right of each row are the group size, the printed estimate and interval, and the corrected one. All respondents, n = 1384: favour universal background checks, 77 (75-79) and 77 (74-80); prefer stricter gun laws, 53 (51-56) and 53 (50-57); believe checks are already universal, 41 (38-43) and 41 (37-45); say checks cover only some purchases, 47 (44-50) and 47 (43-51); believe no checks are required, 12 (10-14) and 12 (10-15). Strong supporters of universal checks, share preferring stricter gun laws: believe checks are already universal, n = 306, 74 (69-79) and 74 (67-81); know some sales are exempt, n = 334, 89 (86-92) and 89 (85-93). Share who would never vote for someone with a different position on gun control: want less strict gun laws, n = 191, 71 (65-78) and 71 (63-80); want stricter gun laws, n = 746, 34 (30-37) and 34 (29-39). The widest blue interval, for n = 191, reaches about 8.6 points on each side of the estimate, against about 6.4 for the orange one. — The nine 95% intervals, each drawn relative to its estimate. Orange: $\hat p\pm1.96\sqrt{V_0}$ , which rounds to the printed interval in every row. Blue: $\hat p\pm1.96\sqrt{V}$ , the stated independent-respondent survey linearisation allowing for the weights. Orange reconstructs the printed arithmetic without identifying the historical method; in the stricter-law row, the unrounded blue lower endpoint is $49.51388\%$ . The columns at the right give the size of the group, the estimate and interval as printed, and the corrected interval, in per cent.

My comment and the history of this entry

I agree that this is a preferable analysis. The original analysis was just performed via weighted least squares (just intercept-only regression in R) with classical standard errors. But it doesn't materially affect any conclusions, and so I think this is minor.

Suggested classification: Minor error.

My position My comment above; I suggested a Minor error.

The models’ position Under the stated survey reconstruction, every interval widens substantially, and the interval for stricter laws no longer excludes 50% at the two-sided 95% level: Moderate.

Classification before my review (the classification when the audit was first completed, before I reviewed it): Moderate error.
I commented and suggested a Minor error.
Independent evaluation. Incorporated in part. The entry records my report that the original analysis used intercept-only weighted least squares in R with classical standard errors; this reproduces the rounded numbers but could not be independently identified as the original procedure. My suggested rating (Minor) was not adopted.