REGISTERED REPORTS AS A METHOD TO INCREASE CREDIBILITY OF SCIENCE – EXPERIMENTAL STUDY AMONG PSYCHOLOGICAL STUDENTS
Aims: The modern scientific publishing system suffers from many problems, amongst which one of the most important is the pressure to publish positive results. A potentially simple way to mitigate publication bias are reviews of manuscript, prior to the data collection and analyses, as well as results presentation and discussion (i.e. Registered Reports).
Methods: To test this prediction we conducted a quasi-experiment: two groups of students (n=38), as a part of their academic classes, were asked to design and conduct research projects. They were divided in two groups – Registered Reports- and control group. In both groups students have been encouraged to publish their papers in a local scientific journal.
Results: Analyses revealed significant differences in p levels between groups, suggesting more reliable scores for Registered Reports group.
Conclusions: Our study partially confirmed the stated hypothesis and suggested, that registered reports might mitigate publication bias. Future recommendations are advised.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a
Bem, S. L. (1974). The measurement of psychological androgyny. Journal of consulting and clinical psychology, 42(2), 155.
Callaway, E. (2011). Report finds massive fraud at Dutch universities. Nature, 479(7371), 15.
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Altmejd, A. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637.
Enserink, M. (2012). Final Report on Stapel Also Blames Field As a Whole. Science, 338(6112), 1270–1271. https://doi.org/10.1126/science.338.6112.1270
Erlich, A. (2018). Pre-acceptance as a method to combat publication bias in area studies: a pilot in the Caucasus. Caucasus Survey, 6(3), 224-229.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738.
Fanelli, D. (2010). “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068
Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 49(13), 1667-1703.
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The Extent and Consequences of PHacking in Science. PLOS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106
Jackson, C. J., Levine, S. Z., Furnham, A., & Burr, N. (2002). Predictors of Cheating Behavior at a University: A Lesson From the Psychology of Work. Journal of Applied Social Psychology, 32(5), 1031–1046. https://doi.org/10.1111/j.1559-1816.2002.tb00254.x
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kuczyńska, A. (1992). Inwentarz do oceny płci psychologicznej. Podręcznik. [The Psychological Sex Inventory] Warszawa: Pracownia Testów Psychologicznych PTP.
Lakens, D. (2015). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. https://doi.org/10.7717/peerj.1142
Leimu, R., & Koricheva, J. (2004). Cumulative meta–analysis: a new tool for detection of temporal trends and publication bias in ecology. Proceedings of the Royal Society of London. Series B: Biological Sciences, 271(1551), 1961-1966.
Levine, T. R., Asada, K. J., & Carpenter, C. (2009). Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings. Communication Monographs, 76(3), 286–302. https://doi.org/10.1080/03637750903074685
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542.
Morgan, G. A. (2000). Quasi-Experimental Designs. Journal of the American Academy of Child & Adolescent Psychiatry. 39. 794–796
McCabe, D. L., Treviño, L. K., & Butterfield, K. D. (2001). Cheating in academic institutions: A decade of research. Ethics & Behavior, 11(3), 219–232.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., Simonsohn U., Wagenmakers E.J., Ware J.J. & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 21. https://doi.org/10.1038/s41562-016-0021
Nosek, B. A., & Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600-2606
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058
Oleszkiewicz, A., Karwowski, M., Pisanski, K., Sorokowski, P., Sobrado, B., & Sorokowska, A. (2017a). Who uses emoticons? Data from 86 702 Facebook users. Personality and Individual Differences, 119, 289-295.
Oleszkiewicz, A., Frackowiak, T., Sorokowska, A., & Sorokowski, P. (2017b). Children can accurately recognize facial emotions from emoticons. Computers in Human Behavior, 76, 372-377.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716
Powell, K. (2016, November 4). Hard work, little reward: Nature readers reveal working hours and research challenges. Retrieved form http://www.nature.com/news/hard-work-little-reward-nature-readers-reveal-working-hours-and-research-challenges-1.20933
Probst, T. M., & Hagger, M. S. (2015). Advancing the Rigour and Integrity of Our Science: The Registered Reports Initiative: Editorial. Stress and Health, 31(3), 177–179. https://doi.org/10.1002/smi.2645
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
Saunders, R., & Savulescu, J. (2008). Research ethics and lessons from Hwanggate: what can we learn from the Korean cloning fraud? Journal of Medical Ethics, 34(3), 214–221. https://doi.org/10.1136/jme.2007.023721
da Silva, J. A. T. (2016). Does the removal of results from a submitted paper reduce publication bias?. Pacific Science Review B: Humanities and Social Sciences, 2(1), 29-30
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Simonsohn, U. (2013). Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone. Psychological Science, 24(10), 1875–1888.https://doi.org/10.1177/0956797613480366
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014a). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014b). p-Curve and effect size correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666–681.
Sorokowski, P., Kulczycki, E., Sorokowska A., Pisanski, K. (2017). Predatory journals recruit fake editor. Nature, 543, 481-483.
Stahl, D., & Pickles, A. (2018). Fact or fiction: reducing the proportion and impact of false positives. Psychological Medicine, 48(7), 1084-1091
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49(1), 108-112.
Stroebe, W., & Strack, F. (2014). The Alleged Crisis and the Illusion of Exact Replication. Perspectives on Psychological Science, 9(1), 59–71. https://doi.org/10.1177/1745691613514450
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science, 4(3), 274–290. https://doi.org/10.1111/j.1745-6924.2009.01125.x
Wagenmakers, E., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results. PLoS ONE, 6(11), e26828. https://doi.org/10.1371/journal.pone.0026828
Wicherts, J., Veldkamp, C., Augusteijn, H., Bakker, M., van Aert, R., & van Assen, M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2017). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25, 1968-1972
Copyright (c) 2019 Piotr Sorokowski, Agata Groyecka, Kamil Błaszczyński, Tomasz Frąckowiak, Aleksander Kobylarek
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. All authors agree for publishing their email adresses, affiliations and short bio statements with their articles during the submission process.