Lernzettel: Enhancing Scientific Self-Correction

📋 Course Outline

1. Myth of self-correcting science
2. Retractions and honest errors
3. Types of honest errors
4. Error detection techniques
5. Statcheck and reporting inconsistencies
6. GRIM, randomization and blinding checks
7. Measurement practices and statistical power
8. Data anomalies and self-correction
9. Global solutions and good practices

📖 1. Myth of self-correcting science

🔑 Key Concepts & Definitions

• Error-prone humans : A description of how people inevitably make mistakes in thinking, measurement, and data handling.
• Self-correction myth : The belief that scientific progress automatically corrects false or flawed findings without deliberate action.
• Loss-of-Confidence survey : A survey of psychology researchers about barriers that prevent collective correction after loss of confidence in findings.

📝 Essential Points

• Science does not self-correct by default, so researchers must actively challenge and repair the record.
• A survey reported 44% of respondents lost confidence in at least one finding.
• Among those, 56% attributed the loss to researchers’ mistake or judgment shortcomings, and 28% took primary responsibility for the error.

💡 Memory Hook

Self-correction is not automatic: confidence loss must trigger action.

📖 2. Retractions and honest errors

🔑 Key Concepts & Definitions

• Retraction : A journal notice that withdraws a published record because of problems affecting its validity.
• Retract and Replace : A correction pathway where an item’s status is changed to non-citable and the journal republishes after substantial changes.
• Retraction Watch : A retraction monitoring source that tracks notices and the stated reasons behind them.

📝 Essential Points

• Retractions can result from misconduct and from issues such as data, results, methods, images, or text.
• Honest errors can also lead to retractions, even without dishonesty like fabrication.
• Self-initiated corrections after honest errors can produce neutral or improved reputation outcomes in some cases.

💡 Memory Hook

Retraction can be for misconduct or for honest error; correction can still be value-positive.

📖 3. Types of honest errors

🔑 Key Concepts & Definitions

• Careless errors : Mistakes arising from inattention or oversight such as typos, copy/paste mistakes, or code errors.
• Negligent errors : Problems caused by incompetence or poor training, including weak design and analysis choices.
• QRP : Questionable research practices that can degrade integrity, often grouping together with measurement issues like QMPs.

📝 Essential Points

• Careless errors include numerical mistakes, typos in data entry, coding errors, and rounding or copy/paste errors.
• Negligent errors include research design and analysis failures, including power or sample-size problems and measure selection problems.
• A noted example is the Reinhart and Rogoff case where an Excel omission removed a country’s debt, eliminating the reported main effect.

💡 Memory Hook

Careless = slips; Negligent = inadequate competence/training leading to flawed design or analysis.

📖 4. Error detection techniques

🔑 Key Concepts & Definitions

• Error detection : A set of methods used to identify inconsistencies, anomalies, or signs of mistakes in scientific reports and results.
• Statcheck : A statistics-focused inconsistency checker that recalculates p-values from reported test statistics and related ingredients.
• GRIM test : A reporting anomaly check that tests whether reported means are compatible with the granularity of the underlying data.

📝 Essential Points

• Low-cost detection tools can use summary information like p-values, t-values, means, sample sizes, and SDs from papers.
• Statcheck works like a spellcheck for statistics because many values in tests depend on each other.
• Statcheck can miss issues when results are reported in an incompatible way or when ingredients needed for recalculation are missing.

💡 Memory Hook

Detection often compares what you see (reported summaries) against what must be true (statistical dependencies).

📖 5. Statcheck and reporting inconsistencies

🔑 Key Concepts & Definitions

• Statcheck inconsistencies : Mismatches that arise when reported statistics, degrees of freedom, and p-values do not jointly match a valid recalculation.
• Optimistic rounding : A reporting practice where borderline p-values are rounded in a way that can shift interpretation toward significance.
• Granularity incompatibility : A situation where the reported mean cannot arise from the stated number of possible discrete response values.

📝 Essential Points

• Statcheck highlights issues including copy/paste errors, wrong inequality signs (>, <), and using = instead of <.
• Optimistic rounding can convert borderline p-values such as p ≥ 0.055 into p < 0.05.
• In a dataset of about 30K studies (1985–2013), 50% had at least one inconsistency and 13% had inconsistencies that could flip significance in conclusions.

💡 Memory Hook

Statcheck catches “math mismatch,” not the underlying reason for an error.

📖 6. GRIM, randomization and blinding checks

🔑 Key Concepts & Definitions

• GRIM granularity : A principle that the format of a mean (decimal endings) must be consistent with the discrete granularity of the data.
• Randomisation check : A test of whether baseline distributions in randomized trials look plausible under a randomization process.
• Blinding check : A check ensuring experimenters collecting data and/or running analyses are unaware of hypotheses and/or group assignments.

📝 Essential Points

• GRIM flags impossible means, for example averages derived from whole-number responses can only end in .00 or .50.
• A randomisation check tests how likely reported baseline points are under random assignment to groups.
• In anesthetics research, papers that used randomization and blinding correctly found smaller effects, and only 30% of animal research papers reported doing it.

💡 Memory Hook

GRIM checks number-format plausibility; randomization and blinding checks check assignment plausibility and separation of knowledge.

📖 7. Measurement practices and statistical power

🔑 Key Concepts & Definitions

• Questionable measurement practices : Measurement approaches that weaken construct validity, such as ad hoc item creation or mixing items across scales.
• Statistical power : The ability of a study design to reliably detect an effect of a specified size given sampling variability.
• Low power : A situation where the study can mainly detect only large effects, making results more vulnerable to exaggeration and error.

📝 Essential Points

• Many Labs 2 found measures were often short and had limited validity evidence reported from the original study.
• In Many Labs 2, 79% of item-based scales appeared ad hoc based on the reported materials.
• With n = 20, examples illustrate limited reliability (e.g., men vs women height detected at n = 6, but men vs women weight requires n = 46).

💡 Memory Hook

Low power means you mostly find what is big, exaggerated, or spurious.

📖 8. Data anomalies and self-correction

🔑 Key Concepts & Definitions

• Data anomalies : Reported patterns or summary statistics that are implausible given known constraints of data ranges, granularity, or raw-data structure.
• Self-correction : Steps taken by researchers or authors to update, retract, or replace parts of the record when errors are discovered.
• Self-correction via retraction actions : Correction pathways that involve retraction of reported results followed by updates or replacements to restore accuracy.

📝 Essential Points

• Odd similarities can trigger suspicion, such as when multiple panels or studies share implausibly close patterns across groups or graders.
• A self-correction example described adding or subtracting integers to F values to make results look statistically significant, later leading to retractions of the affected papers.
• Data can be inconsistent with reported means and SDs under known age ranges, making constraints a tool for anomaly detection.

💡 Memory Hook

If the constraints of possible raw data are violated, the summary is suspect.

📖 9. Global solutions and good practices

🔑 Key Concepts & Definitions

• System of checks and balances : A multi-layer approach that distributes responsibility for verifying results across the research workflow and community.
• Good practices for data analysis : Workflow habits such as plotting data to surface issues and ensuring analyses align with transparent reporting.
• Open peer review : A review process where evaluation can be made more transparent, supporting scrutiny and accountability.

📝 Essential Points

• Global solutions include checks and balances, data sharing, pre-prints, open peer review, and full audit procedures.
• Plotting the data is listed as a practical way to detect problems during analysis.
• The overarching conclusion is that open science needs active correction because the literature is not self-correcting by default.

💡 Memory Hook

Make correction easier: share data, expose review, and run audit-style checks.

📅 Key Dates

Date	Event
2012	Ioannidis discusses how unchallenged fallacies can dominate circulating evidence
2013	Life after P-hacking presents examples about detectability and power
2015	Statcheck is introduced (Nuijten et al.)

⚠️ Common Pitfalls & Confusions

1. Students may think more studies automatically improve truth even when errors persist and replication failures are hard to publish.
2. Students may treat every retraction as misconduct, even though honest errors in data, analyses, methods, or text can also trigger retractions.
3. Students may assume one tool (e.g., statcheck) reveals the reason for error, even though it mainly flags statistical reporting inconsistencies.
4. Students may confuse low power with “no effect,” forgetting that underpowered studies are prone to exaggerated or spurious findings.
5. Students may trust reported summary statistics that violate basic constraints like discrete granularity or feasible ranges of raw data.
6. Students may focus only on fraud detection, neglecting that careless and negligent honest errors also distort the scientific record.

✅ Exam Checklist

1. Explain why self-correction does not happen automatically and state what survey evidence shows about barriers and confidence loss.
2. Differentiate misconduct-related retractions from retractions caused by honest errors affecting data, results, methods, images, or text.
3. Classify honest errors as careless versus negligent and list the types of problems each category includes.
4. Give at least two examples of careless error sources and at least two categories of negligent design/analysis failures.
5. Describe what statcheck does conceptually and list several kinds of reporting inconsistencies it can flag.
6. State the requirements statcheck needs to recalculate a p-value and explain at least one reason it may miss an error.
7. Explain the GRIM idea of checking whether a mean is compatible with granularity and give the .00/.50 example.
8. Describe how randomization checks evaluate baseline plausibility and how blinding affects collected data and analysis awareness.
9. List key measurement issues tied to questionable measurement practices and provide the Many Labs 2 statistics about validity evidence and ad hoc scales.
10. Define statistical power and low power, then use the n=20 examples to explain why small samples miss or mislead.
11. Explain what kinds of data anomalies can be detected via constraints, including mean/SD feasibility or odd similarities.
12. State the package of global solutions (checks and balances, data sharing, pre-prints, open peer review, full audit procedures) and at least one good practice for analysis like plotting data.