Reproducibility

What Is Reproducibility?

Reproducibility refers to the ability to obtain consistent results when scientific studies are repeated or their analyses are revisited by other researchers. Put simply, a finding is considered reproducible if other scientists — by following the same methods and conditions, or by analyzing the same data using the same code — are able to reach equivalent conclusions.

This concept lies at the core of scientific reliability: robust results should persist beyond the original study. In recent years, several meta-research studies have revealed a “reproducibility crisis” across multiple fields of knowledge. A significant number of published studies have reported results that could not be confirmed in independent replication attempts.

In psychology and other social sciences, for instance, there has been a surge in failed attempts to replicate effects previously considered solid. This has raised concerns about the trustworthiness and generalizability of scientific evidence. In response, numerous initiatives have emerged to reform scientific practices within the broader movement of Open Science, aiming to enhance transparency and methodological rigor, and thereby address the crisis of confidence and improve study reproducibility.

In this context, reproducibility can be understood along different dimensions, all grounded in the fundamental principle that the same conclusions “would be” or “have been” reached in a reproducible experiment. Simkus et al. (2025) propose a five-type classification:

• Type A Reproducibility (Analytical/Computational): obtaining the same results using the same data, methods, and code from the original study. This type emphasizes the importance of clear and accessible documentation to allow others to verify published results or conduct alternative analyses.
• Type B Reproducibility (Analytical Robustness): reaching the same conclusion using the same data but applying different statistical methods. This type assesses the robustness of findings across various analytical approaches.
• Type C Reproducibility (Repeatability/Same Team): obtaining the same conclusion with newly collected data in a new study conducted by the same team, in the same laboratory, and using the same method.
• Type D Reproducibility (Replicability/Different Team): obtaining the same conclusion with new data from an independent study conducted by a different team in a different laboratory, using the same method as the original study.
• Type E Reproducibility (Generalizability): obtaining the same conclusion with new data from a study that uses different experimental or analytical methods. This type tests the validity of findings under altered conditions, such as different populations or contexts.

It is important to note that the terms reproducibility and replicability are sometimes used with distinct meanings. Some authors define reproducibility as the repetition of analyses using the same data (e.g., reproducing the results of a paper using the original dataset and code), while replicability refers to repeating the study with new data to verify whether the phenomenon holds.

In this text, we adopt a broad definition: reproducibility encompasses both the reproduction of analyses (same data, same methods) and the replication of the study (new data under the same design). Both practices are essential for validating scientific findings. In all cases, the central goal is to determine whether results withstand independent scrutiny.

Why Is Reproducibility Important?

Reproducibility is a cornerstone of the scientific method. Reproducible results increase our confidence that observed phenomena are valid and not merely the outcome of chance, bias, or methodological flaws. When studies fail to reproduce previous findings, this is not inherently negative — identifying inconsistencies is, in fact, part of the self-correcting nature of science.

However, high rates of replication failure may signal systemic issues in research practices, such as weak experimental designs, underpowered studies, publication bias (a preference for publishing “positive” results), and excessive analytical flexibility (p-hacking, HARKing, and related issues).

In Linguistics — a field historically less accustomed to quantitative experimentation — the discussion around reproducibility has also gained momentum. Recent studies have shown that the adoption of transparency practices in linguistics remains slow. For example, in a sample of 600 papers from the field, fewer than 10% shared their materials, data, or protocols openly; none reported pre-registration; and only 1% mentioned having conducted a replication study (Grieve, 2021). These figures suggest ample room for improvement.

As in other disciplines, linguistics faces the challenge of ensuring that its results are trustworthy and cumulative. Various scholars and scientific organizations have proposed solutions to improve reproducibility in linguistic research, involving both methodological enhancements and cultural shifts.

Among the most commonly cited recommendations are: using more rigorous and appropriate statistical analyses; increasing sample sizes to boost statistical power; and adopting a more open approach throughout all stages of the research process — from hypothesis registration and experimental design to public sharing of data, code, and materials, as well as transparent reporting of results (regardless of whether they are positive or null).

In the next sections, we offer practical guidance for authors interested in conducting reproducibility studies in linguistics. The recommendations are organized in stages — from initial planning (including pre-registration) through execution and dissemination. Following them can help align your work with current best practices, improving both its internal quality and its potential contribution to the field.

1. Study Planning and Initial Pre-registration

Selecting a target study: The first step is to select a published result or study that is relevant and worthy of verification. This may involve an influential finding whose robustness you wish to test, a result with important theoretical implications, or even a contradictory outcome in the existing literature.

Keep in mind that replicating a study is not about hunting for errors, but about clarifying the validity and generalizability of a phenomenon — it is part of the cumulative progress of science. Approach replication with a collaborative spirit, not as a “criminal investigation” of the original research.

Review and design: Carefully examine the original paper, its methods, data, and analyses. Aim to understand all the details necessary to repeat the procedure. Then, develop a complete replication plan, defining a priori the research questions or hypotheses to be tested, the experimental design, sample size and data collection criteria (including a power analysis to ensure statistical adequacy), the dependent and independent variables, and the statistical analysis plan.

The more faithfully the plan reproduces the original study (in the case of a direct replication), the more meaningfully the results can be compared. It is also advisable to anticipate, from the outset, any necessary adaptations — such as linguistic adjustments or changes based on the characteristics of the new sample — and to document all such decisions clearly.

Pre-registration and Registered Reports: Once the study protocol is finalized, register it publicly before any data collection begins. Pre-registration involves submitting your experimental plan to an open-access platform (such as the Open Science Framework, AsPredicted, or similar), making the planned hypotheses and methods publicly available. This practice helps prevent post hoc alterations of the research goals (HARKing) and strengthens the credibility of confirmatory results.

In addition, consider submitting the protocol as a Registered Report to a scientific journal. In this editorial format, the manuscript is peer-reviewed before data collection begins. If approved, the study receives in-principle acceptance for publication, regardless of the results, provided the research is conducted according to the registered plan.

The journal Cadernos de Linguística accepts submissions in this format. Authors are encouraged to first submit a registered report containing the introduction, methodology, and proposed analysis plan. Only after this initial stage is approved by the editorial board should data collection proceed. This ensures that the study is evaluated based on methodological rigor, not on the outcome of the results.

Important: When pre-registering your study, be as detailed as possible. Include inclusion/exclusion criteria, how outliers will be handled, which statistical comparisons are planned, and all variables to be collected — even those not directly tied to the main hypothesis. This level of transparency prevents suspicions of ad hoc variable or condition selection.

A detailed pre-registration demonstrates a serious commitment to the study plan and enhances the reliability of confirmatory analyses.

Methods: The methods section should provide a thorough description of all planned procedures, with enough detail to allow other researchers to reproduce the study exactly. This includes:

• Study procedures: Describe each step of the experiment in detail. Any future deviations from the original plan must be documented and justified in the final report.
• Sample characteristics: Provide a full description of the intended participants, objective inclusion and exclusion criteria, and how technical failures or data errors will be handled. If data substitution is allowed, specify the criteria for doing so. Attention to sample diversity is critical to avoiding WEIRD (Western, Educated, Industrialized, Rich, Democratic) bias. Experimental linguistics, for instance, is heavily skewed toward Indo-European languages, particularly English. In studies aiming for broader generalizability (e.g., Type E Reproducibility), systematic sample heterogenization — varying age, gender, or context of data collection — can be an effective strategy to embrace natural variability and strengthen the robustness of findings.
• Analysis plan: Clearly specify all planned analyses, including preprocessing procedures and corrections for multiple comparisons. List any covariates or regressors, and define any analytical contingencies — that is, decisions that depend on the outcome of earlier results. Only pre-registered analyses should appear in the main results section; exploratory analyses may be reported separately. Whenever possible, involve a statistician or ensure that team members have solid training in data analysis.
• Timeline: Provide an estimated timeline for completing the study and the expected date for submitting the final manuscript.

4. Critical Analysis and Dissemination of Results

Interpreting the findings: After completing the replication, the critical analysis of results should be guided by the objectives established in the preregistered protocol. Avoid rushed judgments or interpretations based on expectations of confirmation. The focus should be on methodological consistency and what the data actually show in comparison to the original study. Broadly speaking, three scenarios may emerge:

• Successful replication: The results consistently confirm the original finding. This reinforces the robustness of the investigated phenomenon. Even so, examine nuances: Was there any difference in effect size? Was the pattern consistent across all subgroups? A replication with greater methodological rigor (e.g., larger sample size, blinding, randomization) may even increase confidence in the validity of the effect. Example: “We replicated effect X using a larger sample and a double-blind protocol, thus strengthening the original evidence.”
• Replication failed to find the effect: The data did not confirm the original result — either due to lack of significance, reversal of the effect, or some other divergent pattern. These findings are equally valuable. Avoid sensational interpretations (“the original study was wrong”) and focus on technical analysis: Could the original effect have been a false positive? Are there differences in design, sample, or sociocultural context that might explain the divergence? Did your replication correct limitations of the original study? A balanced approach enhances the value of a non-replication.
• Partially successful replication: Some aspects were replicated, others were not. The effect may have appeared only in a subgroup or with smaller magnitude. Describe the differences in detail and explore plausible contextual hypotheses — especially relevant in linguistic studies, where cultural, dialectal, or pragmatic variations can impact replicability. Explicitly recognizing such variation helps refine theoretical models and improves our understanding of the sensitivity of effects across different contexts.

Writing the manuscript: When drafting the article, maintain the logical and transparent structure of scientific reporting. The introduction should essentially follow the preregistered version, with minor stylistic adjustments and verb tense shifted to the past. Hypotheses must not be altered or expanded. Studies published after the preregistration may be discussed in the Discussion section, provided they are clearly identified as subsequent developments.

The Results section must report all confirmatory analyses as planned, including exact p-values, effect sizes, and confidence intervals. If a preregistered analysis turned out to be logically flawed or inapplicable, explain this clearly. If additional exploratory analyses are conducted, include them in a separate section, with methodological justification and explicit distinction from the planned analyses. The article’s conclusions must be based solely on the confirmatory results.

Do not omit variables or unexpected results. Selectively excluding data that “didn’t support expectations” distorts scientific interpretation. On the contrary, reporting null or divergent results enhances the credibility of the study. Indicate, case by case, whether each preregistered hypothesis was confirmed. Use descriptive and objective language — avoid evaluative terms such as “the replication failed” or “was a success.” Instead, use formulations like: “We did not observe effect X (p=0.45), while the original study reported p<0.01.” This keeps the focus on the data and avoids unnecessary judgment or blame.

When possible, consider contacting the authors of the original study. Some journals — such as Cadernos de Linguística — offer space for original authors to comment on replications of their work, fostering constructive dialogue and scientific transparency.

Submission and publication: Once the manuscript is complete, submit it to the journal of your choice, ensuring that all open materials (data, scripts, protocols) are accessible via trusted repositories. In the case of Cadernos de Linguística, the peer-review process is open and transparent: reviews are signed, and if the article is accepted, both reviews and author responses are published as supplementary material. This promotes editorial accountability, traceability, and encourages constructive peer critique.

By rigorously following all methodological steps — including preregistration, faithful execution of the plan, transparent documentation, and open sharing — your replication is likely to be positively evaluated, even if the results do not confirm the original study. Cadernos de Linguística and other journals committed to open science do not reject manuscripts based on the “novelty” of findings, but rather on methodological quality and contribution to the cumulative body of knowledge.

Dissemination: After publication, promote the study through academic networks, seminars, and conferences. Share your experience with colleagues and students, and encourage the development of new replications. The practice of reproducibility not only strengthens the accumulation of knowledge, but also promotes a culture of openness and methodological rigor in linguistics — contributing to a more transparent, trustworthy, and self-correcting science.

Recommended Readings

• ALGER, Bradley E. Defense of the Scientific Hypothesis: From Reproducibility Crisis to Big Data. New York: Oxford University Press, 2020. A book defending the role of clear and testable scientific hypotheses amid the reproducibility crisis. It explores how inadequate practices threaten confidence in science, and how hypothesis-driven reasoning, combined with tools like big data and robust statistics, can help restore the credibility of discoveries.
• ATMANSPACHER, Harald; MAASEN, Sabine (Eds.). Reproducibility: Principles, Problems, Practices, and Prospects. Chichester: John Wiley & Sons, 2016. A collection of essays by various authors discussing philosophical and methodological principles of reproducibility, challenges faced in different disciplines, recommended practices, and future prospects for open and reproducible science.
• BAUSELL, R. Barker. The Problem with Science: The Reproducibility Crisis and What to Do About It. New York: Oxford University Press, 2021. A book that dissects the so-called reproducibility crisis, exploring contributing factors such as publication bias, questionable research practices, and lack of rigor. It proposes changes — including preregistration and data sharing — to enhance the reliability of scientific findings.
• BOCHYNSKA, A.; KEEBLE, L.; HALFACRE, C.; CASILLAS, J. V.; CHAMPAGNE, I.; CHEN, K.; RÖTHLISBERGER, M.; BUCHANAN, E. M.; ROETTGER, T. B. Reproducible research practices and transparency across linguistics. Glossa Psycholinguistics, vol. 2, no. 1, 2023. This article assesses the prevalence of transparency practices in 600 linguistics papers over time — open access, data/materials sharing, preregistration, etc. — revealing low adoption rates up to 2018/19. It offers a set of recommendations for researchers, journals, and institutions to promote reproducibility in linguistics.
• BRANDT, M. J.; IJZERMAN, H.; DIJKSTERHUIS, A.; FARACH, F. J.; GELLER, J.; GINER-SOROLLA, R.; GRANGE, J. A.; PERUGINI, M.; SPIES, J. R.; VEER, A. V. The replication recipe: what makes for a convincing replication? Journal of Experimental Social Psychology, vol. 50, 2014. A seminal article offering a practical “recipe” for conducting rigorous replications in social psychology. It defines standardized criteria for convincing replications — such as faithful reproduction of the original method, high statistical power, collaboration with original authors, and preregistration — and discusses how to assess replication outcomes. Useful as a step-by-step guide across disciplines.
• DUDDA, L.; KORMANN, E.; KOZULA, M.; DEVITO, N. J.; KLEBEL, T.; DEWI, A. P. M.; SPIJKER, R.; STEGEMAN, I.; EYNDEN, V. V. D.; ROSS-HELLAUER, T.; LEEFLANG, M. M. G. Open science interventions to improve reproducibility and replicability of research: a scoping review. Royal Society Open Science, vol. 12, 2025. A scoping review of open science interventions aimed at improving reproducibility and replicability. It analyzes 105 studies that empirically evaluated practices such as preregistration, data sharing, null result publication, open peer review, and others — discussing which have evidence of effectiveness and where knowledge gaps remain.
• GRIEVE, Jack. Observation, experimentation, and replication in linguistics. Linguistics, vol. 59, no. 5, 2021. This article discusses the challenges of replication in linguistics, arguing that replication failures may result not just from poor practices but from the inherently social nature of language — since social context varies between studies. It advocates caution in the use of experimental methods and highlights the value of observational approaches.
• PORTE, Graeme; McMANUS, Kevin. Doing Replication Research in Applied Linguistics. New York: Routledge, 2019. A book fully dedicated to conducting replication studies in applied linguistics. It covers identifying suitable target studies, detailed planning (direct, conceptual, and partial replications), execution, analysis, and writing, along with publication advice. A highly accessible and essential read for those wishing to engage in replication research in the language sciences.
• SIMKUS, Andrea; COOLEN-MATURI, Tahani; COOLEN, Frank P. A.; BENDTSEN, Claus. Statistical Perspectives on Reproducibility: Definitions and Challenges. Journal of Statistical Theory and Practice, vol. 19, 2025. This article reviews the literature on reproducibility, defining five types of reproducibility and discussing the causes of low reproducibility across fields, as well as statistical approaches to quantify and improve it.