Reproducibility Measure

A Reproducibility Measure is a similarity measure in agreement between results of measurements of the same measurand carried out by the same operating conditions over a time period, or by different observers or under changed conditions of measurement.

Context(s):
- It is the quality of Reproducible Data,
- It is the quality of a Reproducible Research,
- It is the property of Scientific Evidence.
Example(s):
Counter-Example(s):
- Accuracy,
- Irreproducibility,
- Repeatability,
- Uncertainty
See: Scientific Evidence, Metascience, Measurement, Measurand, Scientific Method.

References

(Wikipedia, 2019) ⇒ https://en.wikipedia.org/wiki/Reproducibility Retrieved:2019-10-19.
- Reproducibility is the closeness of the agreement between the results of measurements of the same measurand carried out with same methodology described in the corresponding scientific evidence (e.g. a publication in a peer-reviewed journal). Reproducibility can also be applied under changed conditions of measurement for the same measurand - to check that the results are not an artefact of the measurement procedures [1] [2]. A related concept is replication, which is the ability to independently achieve non-identical conclusions that are at least similar, when differences in sampling, research procedures and data analysis methods may exist. Reproducibility and replicability together are among the main tools of "the scientific method" — with the concrete expressions of the ideal of such a method varying considerably across research disciplines and fields of study.The study of reproducibility is an important topic in metascience.

(Goodman et al., 2016) ⇒ Steven N. Goodman, Daniele Fanelli and John P. A. Ioannidis (2016). "What does research reproducibility mean?". Science translational medicine, 8(341). [https:doi.org/10.1126/scitranslmed.aaf5027 DOI: 10.1126/scitranslmed.aaf5027]
- QUOTE: Methods reproducibility is meant to capture the original meaning of reproducibility, that is, the ability to implement, as exactly as possible, the experimental and computational procedures, with the same data and tools, to obtain the same results. Results reproducibility refers to what was previously described as “replication,” that is, the production of corroborating results in a new study, having followed the same experimental methods. Inferential reproducibility, not often recognized as a separate concept, is the making of knowledge claims of similar strength from a study replication or reanalysis. This is not identical to results reproducibility, because not all investigators will draw the same conclusions from the same results, or they might make different analytical choices that lead to different inferences from the same data. Here, we explore the definitions and operational complexities of each of these concepts ...

(Keyrouz & Mascagni, 2015) ⇒ Walid Keyrouz, and Michael V. Mascagni (2015). Scientific Software Sustainability: The Numerical Reproducibility Challenge (No. Computational Science & Engineering Software Sustainability and Productivity Challenges (CSESSP Challenges)).
- QUOTE: Experimental reproducibility is a cornerstone of the scientific method. The ease of achieving its counterpart in computing, numerical reproducibility, was one of the core assumptions underpinning the growth of scientific computing over the past several decades to become a powerful tool for scientific inquiry that is now widely considered as the third leg of science. The other core assumption was the deterministic behavior of computer hardware. Unfortunately, these assumptions are currently being challenged by hardware developments over the past several decades as discussed and documented in recent reports and workshops. In this position paper, we are advocating for the following actions: - Redefine numeric reproducibility by considering numeric results as computational measurements and treat them as the equivalent of physical measurements (...)