Commentary/article P048

Corrections to: Gnostic Notes on Temporal Validity

Austin Jang, Molly Offer-Westort, Serena Wang, and P. M. Aronow (2024). Gnostic Notes on Temporal Validity. Research and Politics 11(4).

DOI: 10.1177/20531680241307942
Audited edition: Research and Politics, 2024, 11(4); 6 pages in the audited file
SHA-256 of the audited file: 1f7fe1854e51663255476027b28d3573b19dc2b697fba56cbb4597b4eee5fdd4
Permanent address: https://pmaronow.github.io/llm-audit/works/P048/

Summary

This article, a research note, replies to Munger (2023), who argues that an agnostic analyst cannot produce findings that generalize to the future. I argue in the article that agnosticism is skeptical of assumptions and not free of them, that causal knowledge can be useful when a parameter does not transport, and that decay through adaptation can be studied. The article has no theorem, formula, table or data. 10 entries are classified as errors (10 Minor): slips in quotations, citations and reference entries, and one of grammar. None affects the argument.

What was and was not checked

I read all six pages, with the ten notes and the reference list. The article has no theorem, formula, table, figure, data or code, and it reports no number that could be recomputed. The following were checked and held up.

The statement on p. 2 that covariates are balanced in expectation under uniform random assignment holds in exact enumerations of complete randomization with four units and with five units.
The statement of note 5 that a sharp null hypothesis can be tested in a randomized experiment without structural assumptions holds in exact enumerations with four units and with six units, three of them treated: the randomization test rejects with probability at most its level at every attainable level.
All 29 entries of the reference list are cited in the text, every citation in the text has an entry, and notes 1 to 10 are called out in order.
The passages compared with the published text of Munger (2023) agree with it, apart from the hyphens in P048-N001 and P048-N003: the three premises of the contradiction on p. 2, the words “assumption-free”, the symbol $r$ , the long quotation from its p. 3, the sentence quoted in note 7, and the page numbers 2, 3 and 6.
Stocking (1966, p. 875) calls Boas’s posture “agnostic”, as note 9 says. Lucas (1976) links parameter drift in part to agents who adapt their decision rules, which supports the description of Lucas’s view on p. 3 when “learning” is read broadly.
Holland (1986, p. 948) treats the assumptions of his scientific solutions under the headings “Temporal Stability and Causal Transience” and “Unit Homogeneity”. The “temporal consistency” of p. 2 is the article’s own label, given without quotation marks in a list of examples.

The following were not checked. The quotations from Simon (2001) and their page numbers could not be compared with the published chapter; an archived draft of the chapter contains several of the quoted phrases, with “scientist’s hunting license” for the printed “scientists’ hunting license” and with different pagination. The other quotations from Munger (2023) were not compared with that article. The quotations from the other cited works and the descriptions of them, among them Lin (2013), Rubin (1974), Boas (1920), Goodhart (1984), Dranove et al. (2003) and Recht et al. (2019), were not compared with their sources, and neither were the reference entries other than those for Björkegren et al. (2020), Goodhart (1984), Hastie et al. (2009), Hendrycks et al. (2021) and Moszkowski (2014). The statement about Einstein in note 8 was not checked. The concluding remarks name three technical approaches without citations, and there was nothing to check them against.

The article has no supplement, published correction or replication files. I used public copies for the source comparisons described here. The absence of further corrections is not a finding that the article is correct.

Headline claims and related corrections

Agnosticism in statistics is better understood as assumption-skeptical than assumption-free: even standard analyses of randomized experiments rest on assumptions about the structure of the world (SUTVA: no interference, no hidden versions of treatment), so a definition of agnosticism that excludes assumptions would exclude the study of causal effects, and needing assumptions for temporal validity is not by itself disqualifying.
No numbered result. Abstract and Introduction, p. 1; section 'Agnosticism and causality', p. 2; Notes 3, 4 and 5 (pp. 4–5)
See P048-E003, P048-N001, P048-N003.
Munger's 'unsolved contradiction' (external validity needs knowledge of the target; the target is in the future; the future is unknown) is troubling only if guaranteed transport of a causal parameter is the sole criterion of success. Causal knowledge whose estimated parameter does not literally travel can still be useful, by informing heuristic 'laws of qualitative structure' (Simon) and by establishing local particulars (Boas).
No numbered result. Abstract, p. 1; section 'How knowledge travels', pp. 2–3; Notes 6, 7 and 9
See P048-E005, P048-N003, P048-N007.
Knowledge decay in social science often comes from adaptation and learning by the people studied, which is not opaque to researchers. Lucas's critique and Goodhart's law give constructive, microfoundational accounts of parameter drift, with examples in hospital report cards, machine-learning benchmarks and strategic classification; these dynamics can themselves be studied, and changing parameters need not make the underlying process wholly opaque.
No numbered result. Abstract and Introduction, p. 1; section 'Why knowledge decays', pp. 3–4; Note 10
See P048-E001, P048-E002, P048-N009.
Fast-decaying contexts are not 'too expensive for rigorous causal knowledge': rapid knowledge decay alone does not make rigorous causal research unworthy, because researchers may value results other than stable effect estimates (local dynamics, later uses), scientists differ in what they want to study, and the field cannot know in advance which results will matter later.
No numbered result. Section 'How knowledge travels', p. 3 (paragraphs beginning 'Munger argues that' and 'Munger also asks us'); closing paragraph of 'Why knowledge decays', p. 4; Notes 8 and 9
See P048-N008.
Munger's conclusions may be too pessimistic for practice. The paper finds no contradiction among existing methods for external validity, the institutions of social science and a predictive paradigm; with explicit model beliefs ('a bit of gnosis') useful and reasonable claims can be made about the conditions under which findings generalize, and technical tools exist for studying changing systems (extensions of sensitivity analysis, designs for non-stationary multi-arm problems, distributionally and adversarially robust learning).
No numbered result. Abstract, last sentence; section 'Concluding remarks', p. 4; definition of gnosis in Note 2; rests on the preceding three sections

Corrections

Minor (10)

Real but inconsequential, for example a misprint that does not change a result.

P048-E001 Minor

The reference dated 2021 carries a place and date line for 2025, which should be deleted.

Reference list, p. 5 (PDF p. 5), right column, entry “Hendrycks D, Burns C, Basart S, et al. (2021)”

As printed

Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations, Singapore, Thu Apr 24 – Mon Apr 28th, 2025.

The problem

The entry is dated 2021, and the text on p. 4 cites it as “Hendrycks et al., 2021”, but its last line has the place and dates of a meeting in 2025.

Evidence a check of the printed page

The evidence is a check of the page and of the calendar: 24 and 28 April fall on a Thursday and a Monday in 2025, and on a Saturday and a Wednesday in 2021. I conclude that the printed days are those of 2025. I did not check the place and dates of the 2021 meeting against its proceedings, and the correction omits them.

What survives

The benchmark is identified by its title, authors and year, and the argument on p. 4 about adaptation to benchmarks is unaffected.

Corrected

Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations~~, Singapore, Thu Apr 24 – Mon Apr 28th, 2025~~.

Crop of the four lines of the reference entry Hendrycks D, Burns C, Basart S, et al. (2021) Measuring massive Multitask Language Understanding. In: International Conference on Learning Representations, Singapore, Thu Apr 24 - Mon Apr 28th, 2025. A red outline surrounds Singapore, Thu Apr at the end of the third line and another surrounds 24 - Mon Apr 28th, 2025. in the fourth line. — Reference list, p. 5 (PDF p. 5), right column, entry for Hendrycks et al. (2021). The outlines mark the place and date line ending in 2025.

P048-E002 Minor

The third author of the reference is Samsun Knight, and “Samsun K” should read “Knight S”.

Reference list, p. 5 (PDF p. 5), left column, entry “Björkegren D, Blumenstock JE and Samsun K (2020)”

As printed

Björkegren D, Blumenstock JE and Samsun K (2020) “Training machine learning to anticipate manipulation”. arXiv preprint arXiv:2004.03865.

The problem

The entry lists its authors by surname and initials, and it prints the third author’s given name as the surname and the initial of the surname as the given name.

Evidence a check of the printed page

I base this correction on the arXiv listing for arXiv:2004.03865, whose author line reads “Daniel Björkegren, Joshua E. Blumenstock, Samsun Knight”.

What survives

The arXiv identifier locates the work, and the discussion of strategic adaptation on p. 4 does not depend on the entry.

Corrected

Björkegren D, Blumenstock JE and Knight S (2020) “Training machine learning to anticipate manipulation”. arXiv preprint arXiv:2004.03865.

Crop of the three lines of the reference entry Björkegren D, Blumenstock JE and Samsun K (2020) Training machine learning to anticipate manipulation. arXiv preprint arXiv:2004.03865. A red outline surrounds the name Samsun K. — Reference list, p. 5 (PDF p. 5), left column, last entry. The outline marks “Samsun K”.

P048-E003 Minor

Note 4 quotes the title of Lin (2013) only as far as “experimental” and has no comma after it; the note should cite Lin (2013) without the title.

Note 4, p. 5 (PDF p. 5), left column, first sentence

As printed

Following Lin (2013)’s “Agnostic notes on regression adjustments to experimental” literal belief in assumptions is not requisite to learning from settings in which they are applied:

The problem

The quoted title ends at the adjective “experimental”. The reference list on the same page prints the title as “Agnostic notes on regression adjustments to experimental data: reexamining Freedman’s critique”. The sentence also has no comma between the introductory phrase and “literal belief”.

Evidence a check of the printed page

I compared note 4 with the entry for Lin (2013) in the reference list on the same page.

What survives

The quotation about OLS adjustment that follows in the note is unaffected, as is the passage on p. 2 that the note supports.

Corrected

Following Lin (2013), literal belief in assumptions is not requisite to learning from settings in which they are applied:

Crop of the first four lines of note 4 on page 5. The note begins: Following Lin (2013)'s, then in quotation marks Agnostic notes on regression adjustments to experimental, then literal belief in assumptions is not requisite to learning from settings in which they are applied. A red outline surrounds the word experimental and the closing quotation mark that follows it directly. No comma follows the quotation mark. — Note 4, p. 5 (PDF p. 5), left column, first four lines. The outline marks the end of the quoted title, which stops at “experimental”.

Crop of the three lines of the reference entry Lin W (2013) Agnostic notes on regression adjustments to experimental data: reexamining Freedman's critique. Annals of Applied Statistics 7(1): 295-318. A red outline surrounds the words data: reexamining Freedman's critique. — Reference list, p. 5 (PDF p. 5), right column, entry for Lin (2013). The outline marks the part of the title that note 4 omits.

P048-E004 Minor

The book has three authors and is a second edition: “et al.” should be deleted, and “Vol. 2” should read “2nd ed.”

Reference list, p. 5 (PDF p. 5), right column, entry “Hastie T, Tibshirani R, Friedman JH, et al. (2009)”

As printed

Hastie T, Tibshirani R, Friedman JH, et al. (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer, Vol. 2.

The problem

The entry adds “et al.” after the names of all three authors of the book, and it describes the second edition of the book as a second volume.

Evidence a check of the printed page

The evidence is a public source. The title page of the book, as the publisher hosts it, names Trevor Hastie, Robert Tibshirani and Jerome Friedman and reads “Second Edition”, and the publisher’s bibliographic information for the book gives “Edition number: 2”.

What survives

The book is identifiable by its title, authors and year, and the definition of prediction in note 6 does not depend on the entry.

Corrected

Hastie T, Tibshirani R and Friedman JH (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer.

Crop of the three lines of the reference entry Hastie T, Tibshirani R, Friedman JH, et al. (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer, Vol. 2. A red outline surrounds et al. in the first line and another surrounds Vol. 2. at the end of the third line. — Reference list, p. 5 (PDF p. 5), right column, entry for Hastie, Tibshirani and Friedman (2009). The outlines mark “et al.” and “Vol. 2.”

P048-E005 Minor

“The only criteria … is” should read “the only criterion … is”.

Section “How knowledge travels”, p. 2 (PDF p. 2), right column, first sentence after the numbered list

As printed

We agree that if the only criteria for a successful research program is the guaranteed transportability of a causal parameter, then this contradiction would be troubling.

The problem

The sentence names one condition, the guaranteed transportability of a causal parameter, and its verb is the singular “is”, but its noun is the plural “criteria”.

Evidence a check of the printed page

I checked the printed page.

What survives

The conditional argument of the sentence is unchanged, and nothing else depends on the word.

Corrected

We agree that if the only criterion for a successful research program is the guaranteed transportability of a causal parameter, then this contradiction would be troubling.

Crop of four lines of the right column of page 2: item 3 of the numbered list, We cannot have knowledge of the future, and the first three lines of the next paragraph, which begins We agree that if the only criteria for a successful research program is the guaranteed transportability of a causal parameter. A red outline surrounds the word criteria. — Section “How knowledge travels”, p. 2 (PDF p. 2), right column, after the numbered list. The outline marks “criteria”.

P048-N001 Minor

The short quotation from Munger (2023) should read “zero modeling assumptions”, without a hyphen, as the longer quotation above it does.

Section “Agnosticism and causality”, p. 2 (PDF p. 2), left column, first sentence of the paragraph beginning “To contextualize this stance”

As printed

To contextualize this stance, we first note that standard analyses of RCTs do not make “zero-modeling assumptions.” $^{5}$

The problem

The article quotes the same words of Munger (2023, p. 3) twice in this column. The long quotation at the top of the column has “zero modeling assumptions”. The short quotation has “zero-modeling assumptions”, and the hyphen is not a line-break hyphen, since it falls inside a line.

Evidence a check of the printed page

I base this correction on the page and the published text of Munger (2023), which prints “zero modeling” on its p. 3.

What survives

Both forms are read as “no modeling assumptions”, and the argument of the section is unchanged.

Corrected

To contextualize this stance, we first note that standard analyses of RCTs do not make “zero modeling assumptions.” $^{5}$

Crop of five lines of the left column of page 2. The paragraph that begins To contextualize this stance says that standard analyses of RCTs do not make zero-modeling assumptions, with the last two words in quotation marks and followed by the note number 5. A red outline surrounds the hyphenated word zero-modeling, which stands inside one line. — Section “Agnosticism and causality”, p. 2 (PDF p. 2), left column. The outline marks “zero-modeling” in the short quotation from Munger (2023).

Crop of the first five lines of the left column of page 2. The quotation reads: fully agnostic: they ensure unconfoundedness and positivity (internal validity) through research design, with zero modeling assumptions about the structure of the world, followed by the citation Munger 2023: p. 3. A red outline surrounds the two words zero modeling, which have a space between them and no hyphen. — The same column, lines 1–5 of the page. The outline marks “zero modeling”, printed without a hyphen in the long quotation from Munger (2023: p. 3).

P048-N003 Minor

The quotation from Munger (2023, p. 6) should read “best-case scenario”, with a hyphen.

Section “Why knowledge decays”, p. 3 (PDF p. 3), right column, first sentence

As printed

In formalizing the idea of knowledge decay, Munger provides the concept of a decay rate $r$ , which, even in a “best case scenario… is both positive and unpredictable” (p 6).

The problem

Munger (2023, p. 6) prints “best-case scenario”, and the quotation drops the hyphen. The other passages that were compared with that article agree with it, apart from the hyphen in P048-N001: the three premises of the contradiction on p. 2, the words “assumption-free”, the symbol $r$ , the long quotation from its p. 3, the sentence quoted in note 7, and the page numbers 2, 3 and 6.

Evidence a check of the printed page

I base this correction on a comparison with the published text of Munger (2023), Research & Politics 10(3).

What survives

The quoted words keep their meaning, and the article’s account of Munger’s argument is unchanged.

Corrected

In formalizing the idea of knowledge decay, Munger provides the concept of a decay rate $r$ , which, even in a “best-case scenario… is both positive and unpredictable” (p 6).

Crop of the heading Why knowledge decays and the first five lines of the section in the right column of page 3. The first sentence says that Munger provides the concept of a decay rate r, which, even in a best case scenario, is both positive and unpredictable, with the citation p 6. A red outline surrounds the word best at the end of the second line and another surrounds the word case at the start of the third line. No hyphen follows best. — Section “Why knowledge decays”, p. 3 (PDF p. 3), right column, first lines. The outlines mark “best” and “case”, printed without a hyphen at the end of one line and the start of the next.

P048-N007 Minor

In note 7 the closing quotation mark should come before the page reference “(p 2)”.

Note 7, p. 5 (PDF p. 5), left column

As printed

Munger acknowledges that beyond prediction, “There are other goals, of course, and social science is no stranger to methodological pluralism (p 2).”

The problem

The closing quotation mark follows the page reference and the full stop, and the reference is printed inside the quotation. On p. 3 the same construction closes the quotation first: “… is both positive and unpredictable” (p 6).

Evidence a check of the printed page

I checked the printed page.

What survives

The quotation is read as ending at “pluralism”, and nothing else changes.

Corrected

Munger acknowledges that beyond prediction, “There are other goals, of course, and social science is no stranger to methodological pluralism” (p 2).

Crop of five lines of the notes on page 5: the last line of note 6, the three lines of note 7 and the first line of note 8. Note 7 reads: Munger acknowledges that beyond prediction, There are other goals, of course, and social science is no stranger to methodological pluralism (p 2). The quotation opens before There and closes after the full stop that follows (p 2). A red outline surrounds (p 2), the full stop and the closing quotation mark. — Note 7, p. 5 (PDF p. 5), left column. The outline marks the page reference “(p 2).” and the closing quotation mark printed after it.

P048-N008 Minor

The citation of Moszkowski in note 8 lacks its parentheses, and the reference entry prints the translator credit as a containing work after “In:” and the second “Einstein” of the title in lower case.

Note 8, last sentence, p. 5 (PDF p. 5), left column; and the entry “Moszkowski A (2014)” in the reference list, p. 5, right column

As printed

Note 8. Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy Moszkowski (1921/2014).

Reference list. Moszkowski A (2014). Einstein the searcher: his work explained from dialogues with einstein. In: Translated from the Original German by Henry L. Brose. Oxfordshire: Routledge.

The problem

In note 8 the citation is appended to the sentence without the parentheses that a citation outside the grammar of the sentence takes. In the reference entry the translator credit follows “In:” and is set in italics, as the title of a containing work would be, while the title of the book is set in roman, and the second “Einstein” of the title is in lower case.

Evidence a check of the printed page

I checked the printed page.

What survives

The book remains identifiable, and the dates are compatible with an original year paired with the year of the cited edition. The argument about basic science and later innovation does not depend on the form of the citation.

Corrected

Note 8. Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy (Moszkowski, 1921/2014).

Reference list. Moszkowski A (2014). Einstein the searcher: his work explained from dialogues with Einstein. ~~In:~~ Translated from the Original German by Henry L. Brose. Oxfordshire: Routledge.

In the entry the title of the book is set in italics and the translator credit in roman.

Crop of six lines of the notes on page 5: the five lines of note 8 and the first line of note 9. The last sentence of note 8 reads: Einstein himself was emphatically skeptical of the eventual possibility of using the atom to generate energy Moszkowski (1921/2014). A red outline surrounds Moszkowski (1921/2014) and the full stop after it. — Note 8, p. 5 (PDF p. 5), left column. The outline marks the citation “Moszkowski (1921/2014)”, which follows the sentence without enclosing parentheses.

Crop of the three lines of the reference entry Moszkowski A (2014). Einstein the searcher: his work explained from dialogues with einstein. In: Translated from the Original German by Henry L. Brose. Oxfordshire: Routledge. The title is in roman type and the words Translated from the Original German by Henry L. Brose are in italics. A red outline surrounds the passage from einstein, with a lower-case e, through In: to Brose. — Reference list, p. 5 (PDF p. 5), right column, entry for Moszkowski. The outline marks the lower-case “einstein”, the “In:” and the translator credit set in italics.

P048-N009 Minor

The 1984 book Monetary Theory and Practice was published by Macmillan Press in London, not by Springer in Berlin.

Reference list, p. 5 (PDF p. 5), right column, entry “Goodhart CAE (1984)”

As printed

Goodhart CAE (1984) Problems of monetary management: the UK experience. In: Monetary Theory and Practice. Berlin: Springer, 91–121.

The problem

The entry gives Berlin and Springer as the place and publisher of the 1984 book, which was published by Macmillan Press in London. Springer hosts a later reprint.

Evidence a check of the printed page

I base this correction on the front matter of a reprint of the first edition: its title page reads “MACMILLAN PRESS” and “LONDON”, and its copyright page reads “First published 1984 by” and “London and Basingstoke”.

What survives

The entry identifies the chapter and its pages 91–121 correctly, and the discussion of Goodhart’s law on pp. 3–4 does not depend on the imprint.

Corrected

Goodhart CAE (1984) Problems of monetary management: the UK experience. In: Monetary Theory and Practice. London: Macmillan Press, 91–121.

Crop of the three lines of the reference entry Goodhart CAE (1984) Problems of monetary management: the UK experience. In: Monetary Theory and Practice. Berlin: Springer, 91-121. A red outline surrounds Berlin: at the end of the second line and another surrounds Springer, with its comma, at the start of the third line. — Reference list, p. 5 (PDF p. 5), right column, entry for Goodhart (1984). The outlines mark “Berlin:” and “Springer”.

Formal statements (Lean)

Formal results: 0 whole; 0 restricted; 0 refuted; 0 omitted.

Coverage concerns the identified claim under its stated assumptions. A checked proof establishes the formal statement; correspondence to the published text was reviewed by AI. Repository access is currently required to open the source links.

Each result links to its current formal statement and proof. “Whole” covers the identified claim under its assumptions; “restricted” covers the stated narrower scope; “refuted” gives a counterexample to the identified assertion; “omitted” has no current theorem. Any named published theorem used as a premise remains an assumption. Browse the formal results for this work.

A six-page conceptual research note with no theorem, proposition, lemma, displayed formula, proof, estimator definition, table or computation: none of the five headline claims (P048-H1 to P048-H5) is tied to a numbered or formal result, and all ten confirmed errors (all Minor) are bibliographic, quotation-typography or grammatical misprints whose corrections change no mathematical statement, formula or proof step.

Reproduction

What I recalculated or reran from public materials. "Agrees", "Agrees in part" or "Disagrees" compares my result with the printed one. "Not compared" means the comparison could not be made.

Status: Not applicable
Agreement: Not applicable