Book review P050

Corrections to: Review of The Book of Why: The New Science of Cause and Effect

P. M. Aronow and Fredrik Sävje (2020). Review of The Book of Why: The New Science of Cause and Effect. Journal of the American Statistical Association 115(529):482–485.

DOI: 10.1080/01621459.2020.1721245
arXiv: 2003.11635
Audited edition: Journal of the American Statistical Association, 2020, 115(529):482–485; 5 pages in the audited file
SHA-256 of the audited file: 87f5fe686ca28ec5a857793b663ff80a86becc2631e37e77a2329163904d9a2d
Permanent address: https://pmaronow.github.io/llm-audit/works/P050/

Summary

The review assesses The Book of Why by Judea Pearl and Dana Mackenzie. It argues that the book is too optimistic about what causal models can settle, since the central difficulty is the choice of a model, and that its history neglects other approaches to causal inference. The review has no formulas, tables or data. 2 entries are classified as errors (2 Minor): a missing word, and an unmarked change of one word in a repeated quotation. Eight possible errors were not confirmed, and two of them cannot be settled without the book.

What was and was not checked

I read the whole text of the review, both footnotes and the single reference on the published pages. I checked the following and found no error in them. The examples are computed in exact arithmetic.

The limits of identification from a graph (p. 484, left column). On the graph in which an unobserved gene causes both smoking and lung cancer, the effect of smoking is not identified. There exist two models on this graph that have the same observed law and average causal effects $1/8$ and $-1/8$ .
The comparison of Robins' model with Pearl’s (p. 484). There exist three laws of the counterfactuals that have the same observed law and the same law under every static intervention, and whose natural direct effects are $0$ , $1$ and $1/2$ . The cross-world independence that Pearl’s model adds is what identifies the effect, and it has no implication for the observed law or for the law under a static intervention.
Footnote 2, last sentence. A static treatment regime is a fixed intervention, so the do-operator is a special case of g-notation.
Monotonicity and the local average treatment effect (pp. 484–485). With a randomized binary instrument, exclusion, monotonicity and a nonzero first stage, the Wald ratio equals the mean effect among compliers; both are $1/2$ in an example. Without monotonicity the ratio need not be an average effect for any group: with compliers of mass $1/2$ and effect $1$ , and defiers of mass $1/4$ and effect $3$ , it equals $-1$ . A graph records which variables depend on which, and not the sign, shape or strength of the dependence.
Regression discontinuity and difference-in-differences (p. 485). The descriptions of these designs as relying on continuity and on additivity agree with their usual identification conditions. In an example with two groups, two periods and additive untreated means, the difference-in-differences equals the average effect on the treated, $4$ .
Sensitivity analysis (p. 483). In the checked model, outcome risk depends only on a risk-increasing binary confounder, and exposure has no effect. If its prevalences among the unexposed and exposed satisfy $0<p_0\le p_1$ and the unexposed outcome risk is positive, the observed risk ratio is at most $p_1/p_0$ . I derived this bound and checked it on 140 exact cases.
The design-based approach (p. 484, right column). In an example with four fixed units, two of them treated, the difference in means over the six assignments has expectation $3/2$ , the effect in the finite population, and variance $41/12$ , the value of the standard formula. No superpopulation is needed.
Internal consistency. The two goals announced on p. 483 correspond to the two sections that follow, and the back-reference to the costs of agnosticism (p. 484, right column) points to a statement made in the left column of the same page.
Bibliographic details. The check digit of the ISBN 978-0-46-509760-9 is correct. The sole reference, Robins (1986), has its entry in the reference list, and the page given in that entry for the citation, 484, is correct. The pages 482–485 in the journal’s citation line are the pages that the review occupies.

I did not complete the following checks.

The book under review. I did not obtain a complete copy of the cited edition. I read two manuscript chapters posted by Pearl. The manuscript of the introduction supports the wording of the two quotations cited to pp. 19 and 20. The manuscript of Chapter 2 supports the general content of the quotation cited to pp. 79–80 and differs from it in wording. The quotations and page citations to pp. 79–80, 140, 141, 176, 257, 334 and 394 remain unchecked against the published book. Selected typeset pages credit Robins and Heckman, but do not settle whether the book adequately covers the contributions discussed in the review or omits regression discontinuity and difference-in-differences.
Robins (1986), the single reference, which was not read. The description of that paper in the review was checked with exact finite examples and not against the paper, and its priority was not checked.
The historical statements (on Niles and Wright, on Fisher and Cornfield, and on the origins of the design-based approach), which were not compared with their sources.
The statements about consensus in the field and about the popularity of methods in the social sciences, which are judgments.

The review has no proofs, numbered results, tables, figures, supplement, data or replication code. These checks do not show that the review is correct in the respects that were not examined.

Headline claims and related corrections

The book tells a story in which causal inference was in a sorry state until Pearl's 'Causal Revolution' and is now essentially solved, with near-universal consensus on causal diagrams; this presentation may lead newcomers, especially policy-makers, to unrealistically optimistic conclusions and overlooks alternative models.
No numbered result. Opening paragraphs and the statement of the review's two goals, pp. 482-483 (PDF pp. 2-3); quotation from the book p. 19; block quotation from the book p. 141
See P050-N006.
The central problem scientists face is choosing a valid or reasonable causal model, not expressing or analysing one. The book is about causal analysis given a known structure, not causal discovery, so Pearl's approach can clarify where a disagreement lies but cannot adjudicate it; the 1950s smoking-lung cancer debate (Fisher versus Cornfield) illustrates this.
No numbered result. Section 'Unfounded Optimism About Causal Models', p. 483 (PDF p. 3); block quotation from the book pp. 79-80; 'Diagrams 5.1 and 5.2 in the book (p. 176)'
The modes of inquiry that actually settle debates, such as Cornfield's sensitivity analysis and randomized experiments, work because they let scientists stay largely agnostic about the causal structure; the book's treatment of randomized controlled trials (no need to treat them as a gold standard) misses this. Their usefulness comes with limits on what they identify or cover.
No numbered result. p. 483 right column to p. 484 top left (PDF pp. 3-4); block quotation from the book p. 140
Agnostic approaches have limited applicability, but situations where Pearl's framework can be applied are rare, and even with consensus on the structure the calculus will often say inference is impossible: Fisher's genetic confounding was real, a graph cannot encode that it is weak, and 'Pearl's calculus would have told us that progress was not possible until these genes were identified and measured'. Causal inquiry cannot be reduced to a mathematical exercise.
No numbered result. p. 484 left column, second and third paragraphs (PDF p. 4); the smoking-gene example ('as noted in the book')
The book's history of causal inference is selective and narrow. No consensus, not even an emerging one, exists about the superiority of DAGs; several frameworks thrive because scientists face different problems and can defend different assumptions, and Pearl reduces this pluralism to 'cultural resistance'. Notable omissions are the contributions of a large group of scholars and the problem of estimation.
No numbered result. Section 'Pluralism in Causal Inference', p. 484 left column, and footnote 1 (PDF p. 4); 'Whig history'; quotation 'cultural resistance' (p. 394)
Robins (1986) developed a closely related causal model that predates Pearl's. Both are nonparametric structural equation models; the sole distinction is that Pearl's adds independence assumptions, which make it more powerful (natural mediation effects become identified) but have no testable implications, even through experiments. Robins also introduced g-notation and g-computation, which parallel the do-operator and do-calculus.
No numbered result. p. 484, bottom of left column and top of right column, and footnote 2 (PDF p. 4); Robins (1986); quotations from the book p. 20 and p. 334
See P050-E001.
The design-based approach (Neyman, Fisher) treats random assignment as the sole source of randomness and is agnostic about laws of nature and superpopulations; the cost is a circumscribed task: effects of a well-defined intervention for a fixed population in one setting.
No numbered result. p. 484 right column, paragraph beginning 'The next model' (PDF p. 4); cross-reference 'As we noted above'
Pearl's calculus cannot use information about the form of causal relations. The local average treatment effect interpretation of instrumental variables rests on monotonicity, which a DAG cannot encode; instrumental variables, regression discontinuity (continuity) and difference-in-differences (additivity) rely on such assumptions, are immensely popular in the social sciences, and the last two are omitted from the book. Readers who knew this might doubt that causal diagrams give a complete and systematic solution to confounding.
No numbered result. p. 484 right column, last two paragraphs, to the end of the review on p. 485 (PDF pp. 4-5); block quotation from the book p. 257; closing re-quotation of the p. 141 passage
See P050-N006.

Corrections

Minor (2)

Real but inconsequential, for example a misprint that does not change a result.

P050-E001 Minor

The word “out” is missing: the sentence should read “Pearl is quick to point out the first part while neglecting the second.”

p. 484 (PDF p. 4), right column, first (continued) paragraph, third line

As printed

Pearl is quick to point the first part while neglecting the second.

The problem

The phrase “point out” has lost its second word. The preceding sentence supplies the two parts: the additional independence assumptions make Pearl’s model more powerful, and they have no testable implications.

Evidence a check of the printed page

I checked the printed page.

What survives

The meaning is clear from the preceding sentence, and nothing in the review depends on the missing word.

Corrected

Pearl is quick to point out the first part while neglecting the second.

Crop of five lines at the top of the right column of page 484. The third line reads: have testable implications. Pearl is quick to point the first part. A red outline surrounds the two words point the. The fourth line begins: while neglecting the second. — Page 484 (PDF p. 4), top of the right column. The outline marks “point the” in the third line, where the word “out” is missing between the two words.

P050-N006 Minor

The closing sentence quotes p. 141 of the book as “finding a solution”, while the block quotation on p. 483 has “finding that solution”; the change of word is not marked.

p. 485 (PDF p. 5), left column, last sentence of the review; compare the block quotation on p. 483 (PDF p. 3), left column

As printed

Block quotation, p. 483. There is now an almost universal consensus, at least among epidemiologists, philosophers and social scientists, that (1) confounding needs and has a causal solution, and (2) causal diagrams provide a complete and systematic way of finding that solution. (p. 141)

Last sentence, p. 485. … they might question whether “causal diagrams provide a complete and systematic way of finding a solution [to confounding].”

The problem

The review quotes the same sentence of the book twice, and the two quotations differ in one word. In the closing sentence the square brackets mark “to confounding” as an insertion, but “a” stands outside the brackets where the block quotation has “that”. The two quotations cannot both reproduce the wording of the book.

Evidence a check of the printed page

I compared the two printed pages, which show that the quotations differ. I did not read p. 141 of the book, so I cannot establish which quotation matches it.

What survives

The sense of the quotation is unchanged, and the argument of the closing paragraph does not depend on the word. I use the block quotation on p. 483 as the reference wording below and mark the substitution with brackets; the final wording should be compared with p. 141 of the book.

Corrected

… they might question whether “causal diagrams provide a complete and systematic way of finding [a] solution [to confounding].”

Crop of the four lines that end the review on page 485. The last two lines read: question whether, opening quotation mark, causal diagrams provide a complete and systematic way of finding a solution [to confounding], full stop, closing quotation mark. A red outline surrounds the single word a, between finding and solution. — Page 485 (PDF p. 5), left column, the last sentence of the review. The outline marks the word “a” in “finding a solution [to confounding]”.

Crop of the five-line block quotation on page 483. It reads: There is now an almost universal consensus, at least among epidemiologists, philosophers and social scientists, that (1) confounding needs and has a causal solution, and (2) causal diagrams provide a complete and systematic way of finding that solution. (p. 141). A red outline surrounds the word that, at the start of the last line. — Page 483 (PDF p. 3), left column, the block quotation cited to p. 141 of the book, for comparison. The outline marks the word “that” in “finding that solution”.

Formal statements (Lean)

Formal results: 0 whole; 0 restricted; 0 refuted; 0 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

A prose book review (Journal of the American Statistical Association 115:529, printed pp. 482-485, PDF pp. 2-5) with no theorem, formula, identity, bound or proof on any rendered page: its eight headline claims (P050-H1 to P050-H8) are interpretive remarks tied to no numbered result, and its only two confirmed errors, both Minor (P050-E001, the word 'out' missing from 'Pearl is quick to point the first part while neglecting the second', p. 484, PDF p. 4; P050-N006, the closing re-quotation 'finding a solution [to confounding]', p. 485, PDF p. 5, against 'finding that solution', p. 483, PDF p. 3), are a wording misprint and a quotation slip with no mathematical statement to correct.

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status: Not applicable
Agreement: Not applicable