Reproducibility and Replicability

Reproducibility and replicability are different but related concepts. Historically, not all disciplines have considered the need for reproducibility or replicability, however more recently concerns relating to lack of replicability and reproducibility across fields have made the concepts more ubiquitously relevant.

Reproducibility

Reproducibility of research means that the workflow and data used in the research project can be used to yield the same results. For research to be reproducible, a new team of researchers must be able to use the same hypothesis, experimental method, data, population, and general conditions of the first experiment to reproduce the same results.

Replicability

Replicability of research means that when the same workflow (e.g., research question, experimental design or approach, context or population, and analysis plan) is used using a new data set the outcome is consistent (albeit with a small margin of error).

Both reproducibility and replicability are difficult to achieve because they rely on the reported workflow of the original research (which is often not completely transparent).

Source: phdcomics.com, used in: https://coderefinery.github.io/reproducible-research/motivation/

The Replication Crises

Over the past decade, many researchers have become concerned with the “replication crisis“: evidenced by many large-scale studies failing to replicate what were thought to be universal phenomenon. The Nib has produced a visual narrative explaining the replication crisis in Psychology and the impact that it has across fields. There is an ongoing debate on whether a “replication crisis” equally applies to the humanities and social science disciplines as has been found in STEM. While reproducibility may not make sense in many humanities or social science fields, replication of findings, using similar methods and workflows, does. For example, a digital humanities project which uses a script to draw conclusions off of data will not be replicable unless that script is made available along with the publication. Thus, all disciplinary fields are at risk of a replication crisis when non-open workflows are used.

Scenario – Reproducibility

Let’s consider this scenario: you are starting a new collaborative research study and have engaged your partners in a conversation about how to make the results of the study reproducible. One of the project leads, an experienced researcher with many publications under their belt, makes the following comment:

“All of the results will be in the paper, won’t people be able to reproduce our results from there? If they have any more questions they can reach out directly.”

How would you respond to them?

Possible response

You might state that even an extremely detailed description in the method section will not be sufficient in most cases to reproduce it. This could result from several aspects, including: different computational environments, differences in the software versions, contextual information about the sample, software updates, etc. Additionally, spending the time and effort to create a detailed workflow together will increase both the scientific validity of the final results as well as minimize the time required for re-running or extending further studies. Furthermore, depending on when the reproduction takes place, the authors may not be available to contact.

Dig Deeper

Back
Continue

Last updated on December 4, 2024 @12:08 pm