--- http_interactions: - request: method: get uri: https://rogue-scholar.org/api/posts/2b105b29-acbc-4eae-9ff1-368803f36a4d body: encoding: UTF-8 string: '' headers: Connection: - close Host: - rogue-scholar.org User-Agent: - http.rb/5.1.1 response: status: code: 200 message: OK headers: Age: - '0' Cache-Control: - public, max-age=0, must-revalidate Content-Length: - '34600' Content-Type: - application/json; charset=utf-8 Date: - Sun, 18 Jun 2023 15:23:57 GMT Etag: - '"q6g5fpf9uqp4"' Server: - Vercel Strict-Transport-Security: - max-age=63072000 X-Matched-Path: - "/api/posts/[slug]" X-Vercel-Cache: - MISS X-Vercel-Id: - fra1::iad1::sskn7-1687101837449-d00e04e7a361 Connection: - close body: encoding: UTF-8 string: '{"id":"https://doi.org/10.59350/bbcsr-r4b59","uuid":"2b105b29-acbc-4eae-9ff1-368803f36a4d","url":"https://markrubin.substack.com/p/the-preregistration-prescriptiveness","title":"The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science","summary":"Comments on Van Drimmelen (2023)","date_published":"2023-06-06T21:22:00Z","date_modified":null,"date_indexed":"1979-06-17T18:51:53+00:00","authors":[{"url":null,"name":"Mark Rubin"}],"image":"https://substack-post-media.s3.amazonaws.com/public/images/195477de-91ce-4a76-bae8-b6334fb5bcc7_500x379.jpeg","content_html":"

Abstract

I discuss Van Drimmelen’s (2023) Metascience2023 presentation on researchers’ decision making during the research process. In particular, I consider his evidence that researchers’ discretion over research decisions is unavoidable when they follow research plans that are either overdetermined (i.e., too prescriptive) or underdetermined (i.e., too vague). I argue that this evidence points to a prescriptiveness trade-off when writing preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. I also consider Van Drimmelen’s idea that researchers may make unconscious, implicit decisions during the research process. I relate these implicit decisions to Rumsfeld’s (2002) concept of unknown unknowns: “the things we don’t know we don’t know”! I argue that scientists can report their known knowns (what they know they did and found), and they can be transparent and speculative about their known unknowns (what they know they didn’t do and may find), but that they can’t say much about their unknown unknowns (including their unconscious, implicit decisions) because, by definition, they don’t know what they are! Nonetheless, I think that it’s important to acknowledge unknown unknowns in science because doing so helps to contextualise research efforts as being highly tentative and fallible.

Introduction

The videos of the presentations at the Metascience2023 conference were recently made available here. One that caught my eye is by Tom van Drimmelen, titled “Researchers’ decision making: Navigating ambiguity in research practice”:

Here, I summarise Van Drimmelen’s excellent work and share my thoughts on his ideas.

What is Researcher Discretion?

Van Drimmelen (2023) explained that, during the research process, researchers have discretion over numerous decisions that they make about their methodology and analyses. During the data analysis stage, this researcher discretion may be perceived negatively as researcher degrees of freedom that is susceptible to researcher bias and questionable research practices (John et al., 2012; Simmons et al., 2011).

\"\"
Figure 1. Tom van Drimmelen illustrates the numerous decisions that researchers make during the research process.

Researcher Discretion Emerges in Both Over- and Under-Determined Research Plans

In theory, it is possible to reduce and/or reveal researcher discretion during data analyses by preregistering one or more methodological and analytical research paths. However, Van Drimmelen (2023) and colleagues were interested in how researcher discretion operates in practice. To investigate, they conducted an ethnographic study in which they observed researcher discretion in two different research groups. They found that researcher discretion emerged when researchers used two categories of research plan.

1.     Underdetermined plans: Researcher discretion emerged when research plans were not precise enough to determine specific actions for the researchers. In this case, researchers needed to use their own discretion to fill in the gaps of the research plan and figure out how to proceed. Researchers often find themselves in this situation because preregistered research plans are often vague (e.g., Bakker et al., 2020; Heirene et al., 2021; Van den Akker et al., 2023).

2.     Overdetermined plans: Researcher discretion also emerged when there were unforeseen changes in the research situation that made the original research plan undesirable or impossible, necessitating an alternative research approach. Again, this situation is common, because most researchers do not possess “godlike planning abilities” (Navarro, 2020). To attempt to address this issue, researchers can preregister “if…then” contingencies (i.e., decision trees) that accommodate anticipated changes in the research situation (Nosek et al., 2018, p. 2602). In addition, it may be possible to update a preregistered analysis plan prior to the data analysis in order to take account of unanticipated events (Nosek et al., 2019, p. 817). However, it is often the case that unanticipated events arise during data analyses. As Reinhart (2015, p. 95) explained (paraphrasing Helmuth von Moltke), “no analysis plan survives contact with the data.” In these cases, researchers either need to stick to an inadequate preregistered analysis plan or deviate and adapt their approach to address the current situation. The evidence shows that researchers often deviate from their preregistered plans (e.g., Abrams et al., 2020; Claesen et al., 2021; Heirene et al., 2021).

A Prescriptiveness Trade-Off in Preregistered Plans

Van Drimmelen’s (2023) work shows us that researchers are forced to either deviate from overdetermined plans to overcome unforeseen events or fill in the gaps of vague, underdetermined plans. Hence, researcher discretion emerges in the case of both over- and under-determined plans. As Van Drimmelen (2023) put it, “researcher discretion is an integral and unavoidable part of the research practice.”

I think Van Drimmelen’s (2023) evidence also points to a trade-off between the prescriptiveness of preregistered research plans and the feasibility of their implementation in an unpredictable world. As preregistered plans become more precise and prescriptive, there is a greater chance that researchers will need to deviate from them in order to accommodate unforeseen and/or uncontrolled events that occur during the implementation of the plan. For example, researchers are more likely to deviate from a plan to test “300 participants” than they are to deviate from a less prescriptive plan to test “around 300 participants,” because uncontrolled events (e.g., the number of participant exclusions during data analysis) may result in a final sample size that is only close to 300 participants and not exactly 300 participants (e.g., Claesen et al., 2021, pp. 6-7). Consequently, as preregistered plans become more prescriptive, they are more likely to result in deviations that turn their planned confirmatory tests into unplanned exploratory tests.1 Of course, preregistration continues to distinguish between confirmatory and exploratory tests in this situation, and it prevents exploratory tests from being falsely portrayed as confirmatory tests. Nonetheless, it remains the case that, as plans for confirmatory tests become more prescriptive, the actual implementation of these confirmatory tests in the real world becomes less likely.

On the other hand, as preregistered plans become vaguer and less prescriptive, confirmatory hypothesis tests will become not only more feasible to achieve without deviation, but also more susceptible to researcher degrees of freedom and questionable research practices. For example, it is more feasible to test “around 300 participants” than to test exactly “300 participants.” However, the first sampling plan does not prevent the questionable research practice of optional stopping (John et al., 2021; Simmons et al., 2011), whereas the second, more prescriptive plan does.

In summary, researchers need to consider a prescriptiveness trade-off when writing their preregistered plans: All other things being equal, plans that are more prescriptive are more likely to result in deviations that turn their confirmatory tests into exploratory tests, and plans that are less prescriptive are more likely to result in confirmatory tests that are susceptible to questionable research practices. Hence, researchers face a difficult choice between prescriptive confirmatory tests that they are less likely to carry out in practice and vague confirmatory tests that are more prone to questionable research practices.

\"\"

Researchers Sometimes Make Implicit Decisions

Van Drimmelen and colleagues (2023) also identified another interesting issue in their work. At times, they found it difficult to identify researcher discretion because there were multiple potential methodological or analytical approaches available that researchers did not consider. In other words, sometimes researchers were not aware that they were making decisions!

This idea of unconscious and implicit research decisions reminds me of something that Donald Rumsfeld, the United States Secretary of Defence, said at a press briefing in 2002:

There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. (Wikipedia, 2023)

In the context of Figure 1, our known knowns are the path or paths that we reported following in our research (e.g., the green paths). In these cases, we know what we did, and we know what we found as a result. In contrast, our known unknowns are the paths that we actively decided not to follow (e.g., all the other paths in Figure 1). We know these paths exist, but we don’t yet know what results lie at the end of them.

But what about our unknown unknowns? These paths don’t appear in Figure 1! They relate to decisions that we didn’t even consider, at least not at a conscious level. For example, we may not have considered the time of day of testing, the ambient temperature, or a host of other unspecified variables that may be influential on our results. Instead, we unconsciously relegated these unknown unknowns to a ceteris paribus clause (i.e., “all other things being equal”), which assumes that no other influential factors are at play (Meehl, 1990). The problem is that our unconscious and implicit decision to assume the irrelevance of these factors may be wrong! Indeed, the history of science is full of cases in which factors that were not initially considered to be important subsequently turned out to be crucial moderators or boundary conditions of effects (for some examples, see Firestein, 2016).

\"\"

We Can’t Be Transparent About Unknown Unknowns, But We Should Acknowledge They Exist!

Van Drimmelen (2023) concluded that “we might not be able to be effectively transparent because we don’t know which decisions we’ve actually made” (my emphasis). In other words, our lack of awareness about our potentially important implicit decisions means that we can’t be transparent about them: We can report our known knowns, and we can be transparent and speculative about our known unknowns, but we can’t say much about our unknown unknowns because we don’t know what they are! By definition, as soon as we start to speculate on a potentially influential factor in our research, it becomes a known unknown about which we need to make conscious decisions!

Despite our inability to specify our unknown unknowns, Van Drimmelen’s (2023) work reminds us that it’s important to acknowledge this type of ignorance in science because it helps to contextualise our research efforts as being highly tentative and fallible steps in a collective process of discovery and theory development. Acknowledging our unknown unknowns also helps to address recent calls for greater intellectual humility and modesty in science (Hoekstra & Vazire, 2021; Ramsey, 2021).

I’ll end with a nice example of scientists acknowledging unknown unknowns. Here’s Errington et al. (2021, p. 12) discussing the challenges of assessing replicability in preclinical cancer biology:

It might also be that, in some cases, a failure to replicate was caused by the replication team deviating from the protocol in some way that was not recognized, or that a key part of the procedure was left out of the protocol inadvertently. It is also possible that the effect reported in the original paper depended on methodological factors that were not identified by original authors, the replication team, or any other experts involved in the peer review of the original paper or the Registered Report.

References

Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements (No. w27250). National Bureau of Economic Research. https://doi.org/10.3386/w27250

Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A.,…& Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937. https://doi.org/10.1371/journal.pbio.3000937

Claesen, A., Gomes, S., Tuerlinckx, F., Vanpaemel, W., & Leuven, K. U. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 1–11. https://doi.org/10.1098/rsos.211037

Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Reproducibility in cancer biology: Challenges for assessing replicability in preclinical cancer biology. Elife, 10, Article e67995. https://doi.org/10.7554/eLife.67995

Fife, D., & Rodgers, J. L. (2019). Moving beyond the \"replication crisis\": Understanding the exploratory/confirmatory data analysis continuum. PsyArXiv. https://doi.org/10.31234/osf.io/5vfq6

Firestein, S. (2016, February 14). Why failure to replicate findings can actually be good for science. LA Times. https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html

Heirene, R., LaPlante, D., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. M. (2021). Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison. PsyArXiv. https://psyarxiv.com/nj4es/

Hoekstra, R., & Vazire, S. (2021). Aspiring to greater intellectual humility in science. Nature Human Behaviour, 5(12), 1602-1607. https://doi.org/10.1038/s41562-021-01203-8

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108-141. https://doi.org/10.1207/s15327965pli0102_1

Navarro, D. (2020, September 23). Paths in strange spaces: A comment on preregistration. PsycArXiv. https://doi.org/10.31234/osf.io/wxn58

Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ‘t Veer, A. E., & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23, 815–818. https://doi.org/10.1016/j.tics.2019.07.009

Ramsey, R. (2021). A call for greater modesty in psychology and cognitive neuroscience. Collabra: Psychology, 7(1), 24091. https://doi.org/10.1525/collabra.24091

Reinhart, A. (2015). Statistics done wrong: The woefully complete guide. No Starch Press.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177 /0956797611417632

Van den Akker, O., Bakker, M., van Assen, M. A. L. M., Pennington, C. R., Verweij, L., Elsherif, M. M.,…Wicherts, J. M. (2023, May 10). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. PsyArXiv. https://doi.org/10.31222/osf.io/h8xjw

Van Drimmelen, T., Slagboom, N., Reis, R., Bouter, L., Van der Steen, J. (2023, May 9). Researchers’ decision making: Navigating ambiguity in research practice. Metascience2023, Washington, DC. Abstract: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/ Preregistration: https://osf.io/tqwgp/

Wikipedia. (2023, March 16). There are unknown unknowns. Wikipedia. https://en.wikipedia.org/wiki/There_are_unknown_unknowns

Endnotes

1. Here, I define confirmatory tests as planned tests, and exploratory tests as unplanned tests. This conceptualization is consistent with the frequentist rationale for preregistration, according to which unplanned tests undermine assumptions regarding Type I error rates (Nosek et al., 2019, p. 816). I acknowledge that there are other definitions for “confirmatory” and “exploratory”, including definitions that view them as existing at opposite ends of a continuum rather than as discrete categories (e.g., Fife & Rodgers, 2019). According to this continuum perspective, larger deviations from preregistered plans result in “more exploratory” tests than smaller deviations. I don’t subscribe to this continuum perspective, because it is unclear how to formally assess some deviations as being “larger” or “smaller” than others. I agree that research studies can be described as being more confirmatory or exploratory, because they can contain a greater proportion of planned or unplanned tests. However, in my view, the tests themselves can only be either confirmatory or exploratory, because they can only be either planned or unplanned.

Acknowledgements

I’m grateful to Tom van Drimmelen for his comments on an earlier version of this article.

PDF Download

You can download a PDF version of this article here.

Reference

The reference for this article is:

Rubin, M. (2023, June 7). The preregistration prescriptiveness trade-off and unknown unknowns in science: Comments on Van Drimmelen (2023). Critical Metascience: MetaArXiv. https://doi.org/10.31222/osf.io/3t7pc

Please Subscribe and/or Share!

Subscribe now

Share

","tags":[],"language":"en","references":[{"doi":"https://doi.org/10.3386/w27250","key":"ref1"},{"doi":"https://doi.org/10.1371/journal.pbio.3000937","key":"ref2"},{"doi":"https://doi.org/10.1098/rsos.211037","key":"ref3"},{"doi":"https://doi.org/10.7554/elife.67995","key":"ref4"},{"doi":"https://doi.org/10.31234/osf.io/5vfq6","key":"ref5"},{"key":"ref6","url":"https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html"},{"key":"ref7","url":"https://psyarxiv.com/nj4es"},{"doi":"https://doi.org/10.1038/s41562-021-01203-8","key":"ref8"},{"doi":"https://doi.org/10.1177/0956797611430953","key":"ref9"},{"doi":"https://doi.org/10.1207/s15327965pli0102_1","key":"ref10"},{"doi":"https://doi.org/10.31234/osf.io/wxn58","key":"ref11"},{"doi":"https://doi.org/10.1016/j.tics.2019.07.009","key":"ref12"},{"doi":"https://doi.org/10.1525/collabra.24091","key":"ref13"},{"doi":"https://doi.org/10.1177%20/0956797611417632","key":"ref14"},{"doi":"https://doi.org/10.1177","key":"ref15"},{"doi":"https://doi.org/10.31222/osf.io/h8xjw","key":"ref16"},{"key":"ref17","url":"https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice"},{"key":"ref18","url":"https://osf.io/tqwgp"},{"key":"ref19","url":"https://en.wikipedia.org/wiki/There_are_unknown_unknowns"}],"blog_id":"ak4s224","blog":{"id":"ak4s224","title":"Critical Metascience","description":"Critical perspectives in metascience and the philosophy of science relating to open science, science reform, and the replication crisis.","language":"en","favicon":"https://substackcdn.com/image/fetch/w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00187507-ddc5-4baa-9dae-79f9487d8bce_325x325.png","feed_url":"https://markrubin.substack.com/feed","home_page_url":"https://markrubin.substack.com","user_id":"8498eaf6-8c58-4b58-bc15-27eda292b1aa","created_at":"2023-05-31T13:17:43+00:00","indexed_at":"2023-04-20","feed_format":"application/rss+xml","license":"https://creativecommons.org/licenses/by/4.0/legalcode","generator":"Substack","category":"Social Sciences","prefix":"10.59350","modified_at":"2023-06-13T00:05:19+00:00","version":"https://jsonfeed.org/version/1.1","backlog":false,"authors":null}}' recorded_at: Sun, 18 Jun 2023 15:23:57 GMT - request: method: head uri: https://doi.org/10.3386/w27250 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:23:58 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '159' Connection: - close Location: - http://www.nber.org/papers/w27250.pdf Vary: - Accept Expires: - Sun, 18 Jun 2023 16:08:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=lIloG0Vb1bkgFhn7aSlBg7k5l8OVqTGzFGRLPiT9C7j4M%2F4D%2BEKLkKVFGMWuPqm9B0X73HnVug%2F7cK%2BNg%2Fcx6fJAVgsftjVFhW8K1OJsUD0qsNVu7d8RRk8wAtDZkZt1Avec0JY%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d948957cdc4b724-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:23:58 GMT - request: method: head uri: https://doi.org/10.1371/journal.pbio.3000937 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:23:58 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '181' Connection: - close Location: - https://dx.plos.org/10.1371/journal.pbio.3000937 Vary: - Accept Expires: - Sun, 18 Jun 2023 15:25:39 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=45NMfW4s%2B8nDNY%2FaPOJAXg%2Bh7GnEziH6diotYTqGqoOAIl63J4P9AtKNAv1Y10Oym1jk1h1EQeYSreGsccym0STpUkY4TqlyIORTOFvQcDEhrQMLJtYDnh04rq%2BkZ4VuuwL9Ys4%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94895aa95a0bad-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:23:58 GMT - request: method: head uri: https://doi.org/10.1098/rsos.211037 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:23:59 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '201' Connection: - close Location: - https://royalsocietypublishing.org/doi/10.1098/rsos.211037 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:08:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=clyZ22IZly2qbhtbv%2Bd%2FqOJw3BwnQIUl8tcYRom3HsmSpzbQQgjiaS%2BKNJhWnjdtHZ4%2F7i1gFG3O63PPyQQ3tHH%2FJqtghClUL2E9kV2U3yQaNaX8NlPqLSP46Sm%2F0sm2hKfalB8%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94895d3b001cc6-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:23:58 GMT - request: method: head uri: https://doi.org/10.7554/elife.67995 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:23:59 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '165' Connection: - close Location: - https://elifesciences.org/articles/67995 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:08:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=9XPV0jLnb1M%2Fbx7cdfd2CPWjJI3nr6kNcEQi%2FrSOFSFZwYfrEIlPRMIoGRCLnTdXnl%2BQUdZhGA2mlnZqnq8XlrzTzj78xTQTFXg%2FiNQZy9R5M8LZkNgIPArwSkIjpdql7qhDAwE%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94895f0dc71c7a-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:23:59 GMT - request: method: head uri: https://doi.org/10.31234/osf.io/5vfq6 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:23:59 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '125' Connection: - close Location: - https://osf.io/5vfq6 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:01:57 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=lxpjjij6J88vhLeGCUzXuA9J227fEZp4%2FmF9JJNMAedYaNgdrNjCzHhJ0tLLMz9xETFuZX1gE6YzBO31waOUycxaiwiDf9PRq6n4MnMEuQSdD6SjQcI1NSVxSX5WBUh4Qo1352Q%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d9489620eff0a63-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:23:59 GMT - request: method: head uri: https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html body: encoding: UTF-8 string: '' headers: Connection: - close Host: - latimes.com User-Agent: - http.rb/5.1.1 response: status: code: 301 message: Moved Permanently headers: Server: - awselb/2.0 Date: - Sun, 18 Jun 2023 15:24:00 GMT Content-Type: - text/html Content-Length: - '134' Connection: - close Location: - https://www.latimes.com:443/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html body: encoding: ASCII-8BIT string: '' recorded_at: Sun, 18 Jun 2023 15:24:00 GMT - request: method: head uri: https://psyarxiv.com/nj4es body: encoding: UTF-8 string: '' headers: Connection: - close Host: - psyarxiv.com User-Agent: - http.rb/5.1.1 response: status: code: 200 message: OK headers: Date: - Sun, 18 Jun 2023 15:24:01 GMT Content-Type: - text/html Content-Length: - '7071' Connection: - close Vary: - Accept-Encoding - Accept-Encoding Last-Modified: - Tue, 31 Jan 2023 02:21:53 GMT Etag: - '"63d87b41-1b9f"' Cache-Control: - no-cache, no-store, max-age=0, must-revalidate Expires: - Mon, 01 Jan 1990 00:00:00 GMT Pragma: - no-cache Accept-Ranges: - bytes Strict-Transport-Security: - max-age=15724800 body: encoding: ASCII-8BIT string: '' recorded_at: Sun, 18 Jun 2023 15:24:01 GMT - request: method: head uri: https://doi.org/10.1038/s41562-021-01203-8 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:01 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '185' Connection: - close Location: - https://www.nature.com/articles/s41562-021-01203-8 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:08:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=QoaeCZmPX%2B3IYn7IRPxsazkP9GPKjLji0CWsQyLGMGPiGcora3jhl4BHXimDLxUUqWLN60ErLLjWO4%2F9O%2F3L8hVNU2I3nGCuO%2Fkr7ywqq2oUdmc%2B351SfLVszTTXHpWfjpt8Umc%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94896e8bc90dfb-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:01 GMT - request: method: head uri: https://doi.org/10.1177/0956797611430953 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:02 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '197' Connection: - close Location: - http://journals.sagepub.com/doi/10.1177/0956797611430953 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:01:57 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=G2gFZ8y%2BPHvZYD8gseehoRy81w6UOtABkJat2nnzbMJ%2FQXZcq5aYSdsUJkU5RkSCvsMoHKuuJQzu9WhZP%2BYtwnT31jmoVz7ujOwsVKoOPD00v4RBcxdaJDPh9zBcWvWkd4hcCeU%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d9489702d7f012e-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:02 GMT - request: method: head uri: https://doi.org/10.1207/s15327965pli0102_1 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:02 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '207' Connection: - close Location: - http://www.tandfonline.com/doi/abs/10.1207/s15327965pli0102_1 Vary: - Accept Expires: - Sun, 18 Jun 2023 15:56:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=X%2FZQTMtzKC8k4wCsDL28ZQRpp5YVtgAYVr7Heupk6QOl6FO8aPcc2gGQgbRm4TLAbBZiY2h4S3S%2FK71lT%2FB1DIQcxb%2B5ZjCaFTlFEuU8wNcUgutJCTT2hQIhKlYRpLvZRQRc9UY%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d948972a93ab74e-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:02 GMT - request: method: head uri: https://doi.org/10.31234/osf.io/wxn58 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:02 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '125' Connection: - close Location: - https://osf.io/wxn58 Vary: - Accept Expires: - Sun, 18 Jun 2023 15:56:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Bzv7aRIvR0U9fywm6VDolMFguXOF3DudzT9CsQHuK4s%2FpXvWd3DJo96UZxkXew9XRc6ZZAQ%2BVbtAzdO0rumqWAiYjqGj2R%2BbXCex7%2BKe%2FvbEA309vCEiW0Ytzyrsh1el0KeMU9U%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d948974d93d0ba6-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:02 GMT - request: method: head uri: https://doi.org/10.1016/j.tics.2019.07.009 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:03 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '209' Connection: - close Location: - https://linkinghub.elsevier.com/retrieve/pii/S1364661319301846 Vary: - Accept Expires: - Sun, 18 Jun 2023 16:01:57 GMT Link: - "; rel=dul" - "; rel=dul" Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=TMdMYLN0h05h1SDn4lHKIQ5N334TA0g%2BVZ8S5TZpAQCALMxtTaIgxDcOjGFb7LgHO%2F26efoL849q7twi84wXVefn44XbTFP9zCyjCGQKJ%2FpOM%2FvXERroxsrB4Sz%2FIs6rEqg80tk%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d948976fc65b92d-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:03 GMT - request: method: head uri: https://doi.org/10.1525/collabra.24091 body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:03 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '295' Connection: - close Location: - https://online.ucpress.edu/collabra/article/7/1/24091/117222/A-Call-for-Greater-Modesty-in-Psychology-and Vary: - Accept Expires: - Sun, 18 Jun 2023 16:01:57 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=uc5Y7L%2BeYzavGrYFj5C9aYBIwJBYjltleZLrMFY5EyLTzEDkooskpezF3lqYSQWO0GQxfhGD2yOkruPhWzinuqHDODYHGyH%2BAmZLudNTSusenlwJPGld6dZ01Rl1HzFWVd2JWmw%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94897988200a71-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:03 GMT - request: method: head uri: https://doi.org/10.31222/osf.io/h8xjw body: encoding: UTF-8 string: '' headers: Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Sun, 18 Jun 2023 15:24:03 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '125' Connection: - close Location: - https://osf.io/h8xjw Vary: - Accept Expires: - Sun, 18 Jun 2023 16:08:59 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=M5B9atslZU2gzo%2FML%2BNQU5VOf8zAIkRSs5arUhGX4p0cH725stlW8XXNF5mcwZ3VnmVDCBTTl3RwNCPTUhgYpUIsyOqrxuQW3nguT2VM%2FfibJ3oKGkWeO4RS5zfavnkPbBMXv30%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7d94897baad7b8d0-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:03 GMT - request: method: head uri: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice body: encoding: UTF-8 string: '' headers: Connection: - close Host: - metascience.info User-Agent: - http.rb/5.1.1 response: status: code: 301 message: Moved Permanently headers: Date: - Sun, 18 Jun 2023 15:24:04 GMT Server: - Apache Wpo-Cache-Status: - not cached Wpo-Cache-Message: - The request method was not GET (HEAD) X-Redirect-By: - WordPress Upgrade: - h2,h2c Connection: - Upgrade Location: - https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/ Cache-Control: - max-age=7200 Expires: - Sun, 18 Jun 2023 17:24:04 GMT Host-Header: - c2hhcmVkLmJsdWVob3N0LmNvbQ== Content-Type: - text/html; charset=UTF-8 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:05 GMT - request: method: head uri: https://osf.io/tqwgp body: encoding: UTF-8 string: '' headers: Connection: - close Host: - osf.io User-Agent: - http.rb/5.1.1 response: status: code: 308 message: Permanent Redirect headers: Date: - Sun, 18 Jun 2023 15:24:05 GMT Content-Type: - text/html; charset=utf-8 Content-Length: - '249' Connection: - close Location: - https://osf.io/tqwgp/ X-Frame-Options: - SAMEORIGIN X-Cache-Status: - MISS Cache-Control: - no-cache, no-store, max-age=0, must-revalidate Expires: - Mon, 01 Jan 1990 00:00:00 GMT Pragma: - no-cache Strict-Transport-Security: - max-age=15724800 body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:05 GMT - request: method: head uri: https://en.wikipedia.org/wiki/There_are_unknown_unknowns body: encoding: UTF-8 string: '' headers: Connection: - close Host: - en.wikipedia.org User-Agent: - http.rb/5.1.1 response: status: code: 200 message: OK headers: Date: - Sat, 17 Jun 2023 20:55:42 GMT Server: - mw1418.eqiad.wmnet X-Content-Type-Options: - nosniff Content-Language: - en Vary: - Accept-Encoding,Cookie,Authorization Last-Modified: - Thu, 08 Jun 2023 00:03:49 GMT Content-Type: - text/html; charset=UTF-8 Age: - '66504' X-Cache: - cp3056 hit, cp3050 hit/1 X-Cache-Status: - hit-front Server-Timing: - cache;desc="hit-front", host;desc="cp3050" Strict-Transport-Security: - max-age=106384710; includeSubDomains; preload Report-To: - '{ "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }' Nel: - '{ "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}' Set-Cookie: - GeoIP=DE:NW:M__nster:51.96:7.63:v4; Path=/; secure; Domain=.wikipedia.org - NetworkProbeLimit=0.001;Path=/;Secure;Max-Age=3600 - WMF-DP=44b;Path=/;HttpOnly;secure;Expires=Mon, 19 Jun 2023 00:00:00 GMT - WMF-Last-Access-Global=18-Jun-2023;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Thu, 20 Jul 2023 12:00:00 GMT - WMF-Last-Access=18-Jun-2023;Path=/;HttpOnly;secure;Expires=Thu, 20 Jul 2023 12:00:00 GMT X-Client-Ip: - 2001:16b8:1867:2e00:c444:284:a9b0:81f7 Cache-Control: - private, s-maxage=0, max-age=0, must-revalidate Accept-Ranges: - bytes Content-Length: - '194246' Connection: - close body: encoding: UTF-8 string: '' recorded_at: Sun, 18 Jun 2023 15:24:06 GMT - request: method: get uri: https://doi.org/10.3386/w27250 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:50 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '205' Connection: - close Location: - https://api.crossref.org/v1/works/10.3386%2Fw27250/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=1N%2BfY2sxarataoeBrxDf0joRXn6VOt4aQ8H8tGeXLQS7NzeK2oT1Z70J7hS9tKt55MPCyyjt4HCZ6Ae5ozxaWG2fybp%2FOKwLKXa%2FGRZcQHpwh7qB8y%2FPgWQ%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d308fa88f0e6c-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.3386%2Fw27250/transform recorded_at: Wed, 05 Jul 2023 05:25:50 GMT - request: method: get uri: https://doi.org/10.1371/journal.pbio.3000937 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:51 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '233' Connection: - close Location: - https://api.crossref.org/v1/works/10.1371%2Fjournal.pbio.3000937/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=4o%2FlEeXc%2BTLhUCV0qx0frYt2pZEhB1%2F9YarV37HetL491iLrcQRNEOvDOGGDOqsDGzCUmUHrUKMpvjTkNOhmHwFqJSkhSafkd9YwV23S8hy%2BRFN8EqfaGW0%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30923c3db93e-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1371%2Fjournal.pbio.3000937/transform recorded_at: Wed, 05 Jul 2023 05:25:51 GMT - request: method: get uri: https://doi.org/10.1098/rsos.211037 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:51 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '215' Connection: - close Location: - https://api.crossref.org/v1/works/10.1098%2Frsos.211037/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=V70WNwUpjjV%2FfWoiQH%2F%2B57vaYURnx9hyohWsIFdePaZIeJw%2Fld4H0VBXBnRO%2BQt3CPBwO6MgvjmvPVl4C6IusIbDzeG30DRnEoGIsnNP3ObJRZiwGfh9XZo%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d3093dfbab8ca-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1098%2Frsos.211037/transform recorded_at: Wed, 05 Jul 2023 05:25:51 GMT - request: method: get uri: https://doi.org/10.7554/elife.67995 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:51 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '215' Connection: - close Location: - https://api.crossref.org/v1/works/10.7554%2Felife.67995/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=drvUsUH6u%2BlkYsGDha3%2Fz9OFFFngBIOABolTymPiV0fpzpSOKwX6%2Byj1VAMUCuB8sGUeT7HwV1VWnWmtY5iulRvHekrrpeDmycEsAfLmhNwQO77fc7YMibo%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30957e0cb978-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.7554%2Felife.67995/transform recorded_at: Wed, 05 Jul 2023 05:25:51 GMT - request: method: get uri: https://doi.org/10.31234/osf.io/5vfq6 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:52 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '223' Connection: - close Location: - https://api.crossref.org/v1/works/10.31234%2Fosf.io%2F5vfq6/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=vRqcXD6wa3K1q9672307cUmwInW3SKhZGo3ruw1ZXirz3JUu7xJm9nLos%2BVl05vijvrXvH1kSNxhJsZj%2Fe9dfFMy3o39yoOYl9W5j5BZQHM69Gjp4LDqlWM%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30972ae6b7ea-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.31234%2Fosf.io%2F5vfq6/transform recorded_at: Wed, 05 Jul 2023 05:25:52 GMT - request: method: get uri: https://doi.org/10.1038/s41562-021-01203-8 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:52 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '229' Connection: - close Location: - https://api.crossref.org/v1/works/10.1038%2Fs41562-021-01203-8/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:31:20 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=9qYYxKHlKke4%2B%2BhFRuuQkDt3uaCfpW4sXRdypCHgvVucVcOK9YWHUyQOrgqobtOXx4D2FnumZooslq79NMdqJB2oQsl%2BjDZq5Ccl%2FcUvAQt%2B3bDNNJB9ixg%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d3098dac21ca5-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1038%2Fs41562-021-01203-8/transform recorded_at: Wed, 05 Jul 2023 05:25:52 GMT - request: method: get uri: https://doi.org/10.1177/0956797611430953 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:52 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '225' Connection: - close Location: - https://api.crossref.org/v1/works/10.1177%2F0956797611430953/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Pk4Od83yCWYYZwfQ5%2BtlCOmMMMX1%2B%2FYLubBYSpQVmab0oCKNB%2FCwyYUf9FxSDF2QmvBBV%2BBOY7M%2FY5WXceRMVaRzJtlE80%2BKb32yIZkzc18pkrZUBTTH4Ew%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d309b3e071caa-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1177%2F0956797611430953/transform recorded_at: Wed, 05 Jul 2023 05:25:52 GMT - request: method: get uri: https://doi.org/10.1207/s15327965pli0102_1 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:53 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '229' Connection: - close Location: - https://api.crossref.org/v1/works/10.1207%2Fs15327965pli0102_1/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Qt5LY03GSvso9KhUkVUoxt0Ih%2FCd9hE0kxBVeCttDCh0tgOSBjCDg6oGyxfdPfmIzqL9nFhvwEOHyjHayl0yXpGEQHlrj1iYoxnEt1nnHj23nPMBPjN8Bis%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d309dbd131c96-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1207%2Fs15327965pli0102_1/transform recorded_at: Wed, 05 Jul 2023 05:25:53 GMT - request: method: get uri: https://doi.org/10.31234/osf.io/wxn58 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:53 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '223' Connection: - close Location: - https://api.crossref.org/v1/works/10.31234%2Fosf.io%2Fwxn58/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Clvhw8X%2F6eShmOw%2F5SGdPuL8Gdu%2BULg6e2n1VMIH1DLzREbgmZ%2Fbnbcs9v8ro7RejjfW%2BsULZu7eDzUAmUz2BsECCtPSeqgC9HdNnBMYxqGbZcSqb2NzN9k%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30a00cb8b96f-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.31234%2Fosf.io%2Fwxn58/transform recorded_at: Wed, 05 Jul 2023 05:25:53 GMT - request: method: get uri: https://doi.org/10.1016/j.tics.2019.07.009 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:53 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '229' Connection: - close Location: - https://api.crossref.org/v1/works/10.1016%2Fj.tics.2019.07.009/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Link: - "; rel=dul" - "; rel=dul" Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=NqWkyMqUCaZ10auXU0UFBO3YRSiLMi3py1PfhdTssI9qjFs%2Bo%2FfBTSmd6m%2FiZ1AzPAQludOX7VlM26xrEgc1YN078bhY%2Ffddr4pHtxpd5uPiMmp1%2BI26fQk%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30a2cfef0a51-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1016%2Fj.tics.2019.07.009/transform recorded_at: Wed, 05 Jul 2023 05:25:53 GMT - request: method: get uri: https://doi.org/10.1525/collabra.24091 body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:54 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '221' Connection: - close Location: - https://api.crossref.org/v1/works/10.1525%2Fcollabra.24091/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:31:20 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=QcDiFkF539T1nghrLlstNguN4E6UYGjoNLR8RcZunJEZcHkeg4SXSHI1rU%2Fy241t%2FCGU%2B1QJcJuc8LBSm%2Bo5OWXYBdR69RAQNqqErmh59QdoxjrOspD0k8o%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30a47ea10b7b-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.1525%2Fcollabra.24091/transform recorded_at: Wed, 05 Jul 2023 05:25:54 GMT - request: method: get uri: https://doi.org/10.31222/osf.io/h8xjw body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Wed, 05 Jul 2023 05:25:54 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '223' Connection: - close Location: - https://api.crossref.org/v1/works/10.31222%2Fosf.io%2Fh8xjw/transform Vary: - Accept Expires: - Wed, 05 Jul 2023 05:30:31 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=v%2Bq5JX0xVchDq5Kg2qeBU9wR7bUp%2FWm%2FOBRAzBifZAICQvEhCTQXqT1iHd4p5oQntKGaeL%2BBAaxMRp3CDyjzW7owYY2mJmdSV2T2eehbf0rWTYnE51mKWos%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 7e1d30a5bcdb0bde-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |- Handle Redirect https://api.crossref.org/v1/works/10.31222%2Fosf.io%2Fh8xjw/transform recorded_at: Wed, 05 Jul 2023 05:25:54 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.3386%2Fw27250/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:42 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"institution":[{"name":"National Bureau of Economic Research","acronym":["NBER"],"place":["Cambridge, MA"]}],"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:28Z","timestamp":1687134448953},"publisher-location":"Cambridge, MA","reference-count":0,"publisher":"National Bureau of Economic Research","content-domain":{"domain":[],"crossmark-restriction":false},"DOI":"10.3386\/w27250","type":"report","created":{"date-parts":[[2020,6,2]],"date-time":"2020-06-02T01:54:11Z","timestamp":1591062851000},"source":"Crossref","is-referenced-by-count":6,"title":"Research Registries: Facts, Myths, and Possible Improvements","prefix":"10.3386","author":[{"given":"Eliot","family":"Abrams","sequence":"first","affiliation":[]},{"given":"Jonathan","family":"Libgober","sequence":"additional","affiliation":[]},{"given":"John","family":"List","sequence":"additional","affiliation":[]}],"member":"1960","published-online":{"date-parts":[[2020,5]]},"container-title":[],"original-title":[],"deposited":{"date-parts":[[2021,4,16]],"date-time":"2021-04-16T15:57:54Z","timestamp":1618588674000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.nber.org\/papers\/w27250.pdf"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.3386\/w27250","relation":{},"published":{"date-parts":[[2020,5]]}}' recorded_at: Wed, 05 Jul 2023 05:27:42 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1371%2Fjournal.pbio.3000937/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:42 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; rel="item", ; version="vor"; rel="license", ; title="Marjan Bakker"; rel="author", ; title="Elise A. V. Crompvoets"; rel="author", ; title="Courtney K. Soderberg"; rel="author", ; title="Jelte M. Wicherts"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:50Z","timestamp":1687134470935},"reference-count":67,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2020,12,9]],"date-time":"2020-12-09T00:00:00Z","timestamp":1607472000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["726361"]},{"DOI":"10.13039\/100009827","name":"Laura and John Arnold Foundation","doi-asserted-by":"publisher"},{"DOI":"10.13039\/501100011730","name":"Templeton World Charity Foundation","doi-asserted-by":"publisher"},{"DOI":"10.13039\/100000925","name":"John Templeton Foundation","doi-asserted-by":"publisher"},{"DOI":"10.13039\/501100013437","name":"Templeton Religion Trust","doi-asserted-by":"publisher"}],"content-domain":{"domain":["www.plosbiology.org"],"crossmark-restriction":false},"abstract":"Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of \u201cresearcher degrees of freedom\u201d aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called \u201cOSF Preregistration,\u201d http:\/\/osf.io\/prereg\/<\/jats:ext-link>). The Prereg Challenge format was a \u201cstructured\u201d workflow with detailed instructions and an independent review to confirm completeness; the \u201cStandard\u201d format was \u201cunstructured\u201d with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the \u201cstructured\u201d format restricted the opportunistic use of researcher degrees of freedom better (Cliff\u2019s Delta = 0.49) than the \u201cunstructured\u201d format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.<\/jats:p>","DOI":"10.1371\/journal.pbio.3000937","type":"journal-article","created":{"date-parts":[[2020,12,9]],"date-time":"2020-12-09T22:49:35Z","timestamp":1607554175000},"page":"e3000937","update-policy":"http:\/\/dx.doi.org\/10.1371\/journal.pbio.corrections_policy","source":"Crossref","is-referenced-by-count":29,"title":"Ensuring the quality and specificity of preregistrations","prefix":"10.1371","volume":"18","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-9024-337X","authenticated-orcid":true,"given":"Marjan","family":"Bakker","sequence":"first","affiliation":[]},{"given":"Coosje L. S.","family":"Veldkamp","sequence":"additional","affiliation":[]},{"given":"Marcel A. L. M.","family":"van Assen","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-4471-6396","authenticated-orcid":true,"given":"Elise A. V.","family":"Crompvoets","sequence":"additional","affiliation":[]},{"given":"How Hwee","family":"Ong","sequence":"additional","affiliation":[]},{"given":"Brian A.","family":"Nosek","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-1227-7042","authenticated-orcid":true,"given":"Courtney K.","family":"Soderberg","sequence":"additional","affiliation":[]},{"given":"David","family":"Mellor","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-2415-2933","authenticated-orcid":true,"given":"Jelte M.","family":"Wicherts","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2020,12,9]]},"reference":[{"issue":"11","key":"pbio.3000937.ref001","doi-asserted-by":"crossref","first-page":"2600","DOI":"10.1073\/pnas.1708274114","article-title":"The preregistration revolution","volume":"115","author":"BA Nosek","year":"2018","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pbio.3000937.ref002","doi-asserted-by":"crossref","first-page":"632","DOI":"10.1177\/1745691612463078","article-title":"An agenda for purely confirmatory research","volume":"7","author":"EJ Wagenmakers","year":"2012","journal-title":"Perspect Psychol Sci"},{"key":"pbio.3000937.ref003","doi-asserted-by":"crossref","unstructured":"Christensen G, Wang Z, Levy Paluck E, Swanson N, Birke D, Miguel E, et al. Open Science Practices are on the Rise: The State of Social Science (3S) Survey. UC Berkeley: Center for Effective Global Action. 2020. https:\/\/escholarship.org\/uc\/item\/0hx0207r.","DOI":"10.31222\/osf.io\/5rksu"},{"key":"pbio.3000937.ref004","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1056\/NEJMe048225","article-title":"Clinical trial registration: a statement from the International Committee of Medical Journal Editors","volume":"351","author":"C De Angelis","year":"2004","journal-title":"N Engl J Med"},{"issue":"6","key":"pbio.3000937.ref005","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1177\/1745691619858427","article-title":"Curtailing the Use of Preregistration: A Misused Term","volume":"14","author":"DB Rice","year":"2019","journal-title":"Perspect Psychol Sci"},{"issue":"8","key":"pbio.3000937.ref006","doi-asserted-by":"crossref","first-page":"e3081","DOI":"10.1371\/journal.pone.0003081","article-title":"Systematic review of the empirical evidence of study publication bias and outcome reporting bias","volume":"3","author":"K Dwan","year":"2018","journal-title":"PLoS ONE"},{"issue":"8","key":"pbio.3000937.ref007","doi-asserted-by":"crossref","first-page":"e0132382","DOI":"10.1371\/journal.pone.0132382","article-title":"Likelihood of null effects of large NHLBI clinical trials has increased over time","volume":"10","author":"RM Kaplan","year":"2015","journal-title":"PLoS ONE"},{"key":"pbio.3000937.ref008","doi-asserted-by":"crossref","first-page":"c365","DOI":"10.1136\/bmj.c365","article-title":"The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews","volume":"340","author":"JJ Kirkham","year":"2010","journal-title":"BMJ"},{"issue":"2","key":"pbio.3000937.ref009","doi-asserted-by":"crossref","first-page":"e0188247","DOI":"10.1371\/journal.pone.0188247","article-title":"Determinants of selective reporting: A taxonomy based on content analysis of a random selection of the literature","volume":"13","author":"JT van der Steen","year":"2018","journal-title":"PLoS ONE"},{"key":"pbio.3000937.ref010","volume-title":"Thinking, fast and slow","author":"D Kahneman","year":"2011"},{"issue":"2","key":"pbio.3000937.ref011","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1037\/1089-2680.2.2.175","article-title":"Confirmation bias: A ubiquitous phenomenon in many guises","volume":"2","author":"RS Nickerson","year":"1998","journal-title":"Rev Gen Psychol"},{"key":"pbio.3000937.ref012","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.actpsy.2014.02.001","article-title":"The meaning of \"significance\" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969","volume":"148","author":"AD de Groot","year":"2014","journal-title":"Acta Psychol"},{"issue":"6","key":"pbio.3000937.ref013","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1511\/2014.111.460","article-title":"The statistical crisis in science: data-dependent analysis\u2014a \u201cgarden of forking paths\u201d\u2014explains why many statistically significant comparisons don\u2019t hold up","volume":"102","author":"A Gelman","year":"2014","journal-title":"Am Sci"},{"issue":"8","key":"pbio.3000937.ref014","doi-asserted-by":"crossref","first-page":"e124","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"JPA Ioannidis","year":"2005","journal-title":"PLoS Med"},{"issue":"5","key":"pbio.3000937.ref015","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1097\/EDE.0b013e31818131e7","article-title":"Why most discovered true associations are inflated","volume":"19","author":"JPA Ioannidis","year":"2008","journal-title":"Epidemiology"},{"key":"pbio.3000937.ref016","doi-asserted-by":"crossref","first-page":"0021","DOI":"10.1038\/s41562-016-0021","article-title":"A manifesto for reproducible science","volume":"1","author":"MR Munaf\u00f2","year":"2017","journal-title":"Nat Hum Behav"},{"key":"pbio.3000937.ref017","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1177\/0956797611430953","article-title":"Measuring the prevalence of questionable research practices with incentives for truth-telling","volume":"23","author":"LK John","year":"2012","journal-title":"Psychol Sci"},{"key":"pbio.3000937.ref018","doi-asserted-by":"crossref","first-page":"1832","DOI":"10.3389\/fpsyg.2016.01832","article-title":"Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking","volume":"7","author":"JM Wicherts","year":"2016","journal-title":"Front Psychol"},{"issue":"4","key":"pbio.3000937.ref019","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1056\/NEJMsr1601330","article-title":"Update on trial registration 11 years after the ICMJE policy was established","volume":"376","author":"DA Zarin","year":"2017","journal-title":"N Engl J Med"},{"issue":"3","key":"pbio.3000937.ref020","article-title":"Preregistration becoming the norm in psychological science","volume":"31","author":"BA Nosek","year":"2018","journal-title":"APS Obs"},{"issue":"17","key":"pbio.3000937.ref021","doi-asserted-by":"crossref","first-page":"1838","DOI":"10.1001\/jama.2012.3424","article-title":"Characteristics of clinical trials registered in clinicaltrials.gov, 2007\u20132010","volume":"307","author":"RM Califf","year":"2012","journal-title":"JAMA"},{"key":"pbio.3000937.ref022","unstructured":"Simmons JP, Nelson LD, Simonsohn U. How to properly preregister a study [Blog post]. 2017. http:\/\/datacolada.org\/64."},{"key":"pbio.3000937.ref023","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.jesp.2016.03.004","article-title":"Preregistration in social psychology\u2014A discussion and suggested template","volume":"67","author":"AE van\u2019t Veer","year":"2016","journal-title":"J Exp Soc Psychol"},{"key":"pbio.3000937.ref024","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.jesp.2013.10.005","article-title":"The replication recipe: What makes for a convincing replication?","volume":"50","author":"MJ Brandt","year":"2014","journal-title":"J Exp Soc Psychol"},{"issue":"1","key":"pbio.3000937.ref025","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1111\/add.12728","article-title":"Ten reasons why journals must review manuscripts before results are known","volume":"110","author":"CD Chambers","year":"2015","journal-title":"Addiction"},{"key":"pbio.3000937.ref026","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1027\/1864-9335\/a000192","article-title":"Registered reports: A method to increase the credibility of published results","volume":"45","author":"BA Nosek","year":"2014","journal-title":"Soc Psychol"},{"key":"pbio.3000937.ref027","doi-asserted-by":"crossref","unstructured":"Cr\u00fcwell S, Evans NJ. Preregistration in Complex Contexts: A Preregistration Template for the Application of Cognitive Models. 2019. 10.31234\/osf.io\/2hykx","DOI":"10.31234\/osf.io\/2hykx"},{"key":"pbio.3000937.ref028","unstructured":"Flannery J. fMRI Preregistration Template. 2018. https:\/\/osf.io\/dvb2e\/."},{"issue":"3","key":"pbio.3000937.ref029","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1080\/08989621.2019.1580147","article-title":"Preregistering qualitative research","volume":"26","author":"TL Haven","year":"2019","journal-title":"Account Res"},{"issue":"5","key":"pbio.3000937.ref030","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1177\/1745691616658637","article-title":"Increasing transparency through a multiverse analysis","volume":"11","author":"S Steegen","year":"2016","journal-title":"Perspect Psychol Sci"},{"issue":"6","key":"pbio.3000937.ref031","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1037\/a0039191","article-title":"Opportunistic biases: Their origins, effects, and an integrated solution","volume":"70","author":"J DeCoster","year":"2015","journal-title":"Am Psychol"},{"issue":"1","key":"pbio.3000937.ref032","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/s11336-015-9444-2","article-title":"Improving the conduct and reporting of statistical analysis in psychology","volume":"81","author":"K Sijtsma","year":"2015","journal-title":"Psychometrika"},{"key":"pbio.3000937.ref033","doi-asserted-by":"crossref","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"JP Simmons","year":"2011","journal-title":"Psychol Sci"},{"issue":"3","key":"pbio.3000937.ref034","doi-asserted-by":"crossref","first-page":"e0172792","DOI":"10.1371\/journal.pone.0172792","article-title":"Questionable research practices among italian research psychologists","volume":"12","author":"F Agnoli","year":"2017","journal-title":"PLoS ONE"},{"key":"pbio.3000937.ref035","doi-asserted-by":"crossref","first-page":"e5738","DOI":"10.1371\/journal.pone.0005738","article-title":"How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data","volume":"4","author":"D Fanelli","year":"2009","journal-title":"PLoS ONE"},{"issue":"1","key":"pbio.3000937.ref036","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1177\/1948550615612150","article-title":"Questionable research practices revisited","volume":"7","author":"K Fiedler","year":"2016","journal-title":"Soc Psychol Personal Sci"},{"issue":"1","key":"pbio.3000937.ref037","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1177\/1948550615598377","article-title":"Underreporting in Psychology Experiments: Evidence From a Study Registry","volume":"7","author":"A Franco","year":"2016","journal-title":"Soc Psychol Personal Sci"},{"issue":"4","key":"pbio.3000937.ref038","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1177\/1745691613491437","article-title":"PsychDisclosure.org: Grassroot Support for Reforming Reporting Standards in Psychology","volume":"8","author":"E LeBel","year":"2013","journal-title":"Perspect Psychol Sci"},{"issue":"2","key":"pbio.3000937.ref039","first-page":"376","article-title":"The chrysalis effect: How ugly initial results metamorphosize into beautiful articles","volume":"43","author":"EH O\u2019Boyle","year":"2017","journal-title":"J Manag"},{"issue":"7","key":"pbio.3000937.ref040","doi-asserted-by":"crossref","first-page":"e0236079","DOI":"10.1371\/journal.pone.0236079","article-title":"Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size","volume":"15","author":"M Bakker","year":"2020","journal-title":"PLoS ONE"},{"issue":"3","key":"pbio.3000937.ref041","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1037\/met0000014","article-title":"Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: The power of alternatives and recommendations","volume":"19","author":"M Bakker","year":"2014","journal-title":"Psychol Methods"},{"issue":"3","key":"pbio.3000937.ref042","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1037\/0033-2909.114.3.494","article-title":"Dominance statistics: Ordinal analyses to answer ordinal questions","volume":"114","author":"N Cliff","year":"1993","journal-title":"Psychol Bull"},{"key":"pbio.3000937.ref043","volume-title":"Ordinal Methods for Behavior Data Analysis","author":"N Cliff","year":"1996"},{"key":"pbio.3000937.ref044","unstructured":"Romano J, Kromrey JD, Coraggio J, Skowronek J. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen\u2019s d for evaluating group differences on the NSSE and other surveys?. Paper presented at the Annual meeting of the Florida Association of Institutional Research. Chicago, Illinois; 2006."},{"issue":"11","key":"pbio.3000937.ref045","doi-asserted-by":"crossref","first-page":"e230","DOI":"10.1371\/journal.pmed.0050230","article-title":"Bias, spin, and misreporting: time for full access to trial protocols and results","volume":"5","author":"A-W Chan","year":"2008","journal-title":"PLoS Med"},{"issue":"7494","key":"pbio.3000937.ref046","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1136\/bmj.38356.424606.8F","article-title":"Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors","volume":"330","author":"A-W Chan","year":"2005","journal-title":"BMJ"},{"issue":"20","key":"pbio.3000937.ref047","doi-asserted-by":"crossref","first-page":"2457","DOI":"10.1001\/jama.291.20.2457","article-title":"Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles","volume":"291","author":"A-W Chan","year":"2004","journal-title":"JAMA"},{"key":"pbio.3000937.ref048","doi-asserted-by":"crossref","first-page":"a2299","DOI":"10.1136\/bmj.a2299","article-title":"Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols","volume":"337","author":"A-W Chan","year":"2008","journal-title":"BMJ"},{"issue":"7588","key":"pbio.3000937.ref049","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1038\/530007a","article-title":"Make journals report clinical trials properly","volume":"530","author":"B Goldacre","year":"2016","journal-title":"Nature"},{"key":"pbio.3000937.ref050","doi-asserted-by":"crossref","first-page":"j408","DOI":"10.1136\/bmj.j408","article-title":"Outcome reporting bias in clinical trials: why monitoring matters","volume":"356","author":"JPA Ioannidis","year":"2017","journal-title":"BMJ"},{"issue":"4","key":"pbio.3000937.ref051","doi-asserted-by":"crossref","first-page":"1292","DOI":"10.1213\/ANE.0000000000002272","article-title":"Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals","volume":"125","author":"PM Jones","year":"2017","journal-title":"Anesth Analg"},{"issue":"9","key":"pbio.3000937.ref052","doi-asserted-by":"crossref","first-page":"e1232","DOI":"10.1038\/tp.2017.203","article-title":"Outcome reporting bias in randomized-controlled trials investigating antipsychotic drugs","volume":"7","author":"M Lancee","year":"2017","journal-title":"Transl Psychiatry"},{"key":"pbio.3000937.ref053","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.jclinepi.2017.07.014","article-title":"Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy","volume":"91","author":"E Mayo-Wilson","year":"2017","journal-title":"J Clin Epidemiol"},{"issue":"4","key":"pbio.3000937.ref054","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1111\/cob.12199","article-title":"Selective outcome reporting in obesity clinical trials: a cross-sectional review","volume":"7","author":"J Rankin","year":"2017","journal-title":"Clin Obes"},{"issue":"6","key":"pbio.3000937.ref055","doi-asserted-by":"crossref","first-page":"e0178379","DOI":"10.1371\/journal.pone.0178379","article-title":"Evidence of selective reporting bias in hematology journals: A systematic review","volume":"12","author":"C Wayant","year":"2017","journal-title":"PLoS ONE"},{"issue":"49","key":"pbio.3000937.ref056","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1016\/j.cortex.2012.12.016","article-title":"Registered reports: a new publishing initiative at Cortex","volume":"3","author":"CD Chambers","year":"2013","journal-title":"Cortex"},{"key":"pbio.3000937.ref057","volume-title":"Preregistration: Comparing Dream to Reality","author":"A Claesen","year":"2019"},{"issue":"1","key":"pbio.3000937.ref058","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1186\/s13063-019-3173-2","article-title":"COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time","volume":"20","author":"B Goldacre","year":"2019","journal-title":"Trials"},{"issue":"1","key":"pbio.3000937.ref059","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1525\/collabra.71","article-title":"Too good to be false: Nonsignificant results revisited","volume":"3","author":"CH Hartgerink","year":"2017","journal-title":"Collabra"},{"issue":"1","key":"pbio.3000937.ref060","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1037\/pspa0000084","article-title":"The State of Social and Personality Science: Rotten to the Core, Not So Bad, Getting Better, or Getting Worse?","volume":"113","author":"M Motyl","year":"2017","journal-title":"J Pers Soc Psychol"},{"key":"pbio.3000937.ref061","unstructured":"Nelson LD, Simmons JP, Simonsohn U. Forthcoming in JPSP: A Non-Diagnostic Audit of Psychological Research. http:\/\/datacolada.org\/60 2017."},{"issue":"6251","key":"pbio.3000937.ref062","doi-asserted-by":"crossref","DOI":"10.1126\/science.aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"issue":"10","key":"pbio.3000937.ref063","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1016\/j.tics.2019.07.009","article-title":"Preregistration Is Hard, And Worthwhile","volume":"23","author":"BA Nosek","year":"2019","journal-title":"Trends Cogn Sci"},{"key":"pbio.3000937.ref064","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/d41586-019-02674-6","article-title":"What\u2019s next for registered reports?","volume":"573","author":"CD Chambers","year":"2019","journal-title":"Nature"},{"issue":"9","key":"pbio.3000937.ref065","article-title":"Seven selfish reasons for preregistration","volume":"29","author":"EJ Wagenmakers","year":"2016","journal-title":"APS Obs"},{"key":"pbio.3000937.ref066","first-page":"1","article-title":"Study Preregistration: An Evaluation of a Method for Transparent Reporting","volume":"2020","author":"AA Toth","journal-title":"J Bus Psychol"},{"key":"pbio.3000937.ref067","doi-asserted-by":"crossref","first-page":"175","DOI":"10.3758\/BF03193146","article-title":"G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences","volume":"39","author":"F Faul","year":"2007","journal-title":"Behav Res Methods"}],"container-title":"PLOS Biology","original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pbio.3000937","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,9]],"date-time":"2020-12-09T22:50:42Z","timestamp":1607554242000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pbio.3000937"}},"subtitle":[],"editor":[{"given":"Lisa","family":"Bero","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2020,12,9]]},"references-count":67,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,12,9]]}},"URL":"http:\/\/dx.doi.org\/10.1371\/journal.pbio.3000937","relation":{},"ISSN":["1545-7885"],"subject":["General Agricultural and Biological Sciences","General Immunology and Microbiology","General Biochemistry, Genetics and Molecular Biology","General Neuroscience"],"container-title-short":"PLoS Biol","published":{"date-parts":[[2020,12,9]]}}' recorded_at: Wed, 05 Jul 2023 05:27:42 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1098%2Frsos.211037/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:43 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="application/pdf"; rel="item", ; version="vor"; type="application/xml"; rel="item", ; version="vor"; rel="item", ; version="tdm"; rel="license", ; title="Aline Claesen"; rel="author", ; title="Sara Gomes"; rel="author", ; title="Francis Tuerlinckx"; rel="author", ; title="Wolf Vanpaemel"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:56Z","timestamp":1687134476166},"reference-count":30,"publisher":"The Royal Society","issue":"10","license":[{"start":{"date-parts":[[2021,10,1]],"date-time":"2021-10-01T00:00:00Z","timestamp":1633046400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/royalsociety.org\/journals\/ethics-policies\/data-sharing-mining\/"}],"funder":[{"DOI":"10.13039\/501100004040","name":"KU Leuven","doi-asserted-by":"crossref","award":["C14\/19\/054"]}],"content-domain":{"domain":["royalsocietypublishing.org"],"crossmark-restriction":true},"published-print":{"date-parts":[[2021,10]]},"abstract":"\n Preregistration is a method to increase research transparency by documenting research decisions on a public, third-party repository prior to any influence by data. It is becoming increasingly popular in all subfields of psychology and beyond. Adherence to the preregistration plan may not always be feasible and even is not necessarily desirable, but without disclosure of deviations, readers who do not carefully consult the preregistration plan might get the incorrect impression that the study was exactly conducted and reported as planned. In this paper, we have investigated adherence and disclosure of deviations for all articles published with the Preregistered badge in\n Psychological Science<\/jats:italic>\n between February 2015 and November 2017 and shared our findings with the corresponding authors for feedback. Two out of 27 preregistered studies contained no deviations from the preregistration plan. In one study, all deviations were disclosed. Nine studies disclosed none of the deviations. We mainly observed (un)disclosed deviations from the plan regarding the reported sample size, exclusion criteria and statistical analysis. This closer look at preregistrations of the first generation reveals possible hurdles for reporting preregistered studies and provides input for future reporting guidelines. We discuss the results and possible explanations, and provide recommendations for preregistered research.\n <\/jats:p>","DOI":"10.1098\/rsos.211037","type":"journal-article","created":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T07:05:32Z","timestamp":1635318332000},"update-policy":"http:\/\/dx.doi.org\/10.1098\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":"Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies","prefix":"10.1098","volume":"8","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-0303-2441","authenticated-orcid":true,"given":"Aline","family":"Claesen","sequence":"first","affiliation":[{"name":"Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102, Leuven 3000, Belgium"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-2099-6314","authenticated-orcid":true,"given":"Sara","family":"Gomes","sequence":"additional","affiliation":[{"name":"Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102, Leuven 3000, Belgium"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-1775-7654","authenticated-orcid":true,"given":"Francis","family":"Tuerlinckx","sequence":"additional","affiliation":[{"name":"Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102, Leuven 3000, Belgium"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5855-3885","authenticated-orcid":true,"given":"Wolf","family":"Vanpaemel","sequence":"additional","affiliation":[{"name":"Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102, Leuven 3000, Belgium"}]}],"member":"175","published-online":{"date-parts":[[2021,10,27]]},"reference":[{"key":"e_1_3_8_2_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611417632"},{"key":"e_1_3_8_3_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_3_8_4_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2016.01832"},{"key":"e_1_3_8_5_2","unstructured":"Gelman A Loken E. 2013 The garden of forking paths: why multiple comparisons can be a problem even when there is no \u2018fishing expedition\u2019 or \u2018p-hacking\u2019 and the research hypothesis was posited ahead of time. Dep Stat Columbia Univ [Internet]. See http:\/\/stat.columbia.edu\/~gelman\/research\/unpublished\/forking.pdf."},{"key":"e_1_3_8_6_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1708274114"},{"key":"e_1_3_8_7_2","first-page":"aac4716","article-title":"Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","unstructured":"Open Science Collaboration. 2015 Estimating the reproducibility of psychological science. J. Am. Stat. Assoc. 349, aac4716. (doi:10.1126\/science.aac4716)","journal-title":"J. Am. Stat. Assoc."},{"key":"e_1_3_8_8_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0040195"},{"key":"e_1_3_8_9_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918787489"},{"key":"e_1_3_8_10_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612459058"},{"key":"e_1_3_8_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2019.11.009"},{"key":"e_1_3_8_12_2","doi-asserted-by":"publisher","DOI":"10.1177\/0956797613512465"},{"key":"e_1_3_8_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cortex.2012.12.016"},{"key":"e_1_3_8_14_2","unstructured":"DeHaven A. 2017 Preregistration: a plan not a prison [Internet]. [cited 10 February 2021]. See https:\/\/www.cos.io\/blog\/preregistration-plan-not-prison."},{"key":"e_1_3_8_15_2","doi-asserted-by":"publisher","DOI":"10.1111\/1475-679X.12208"},{"key":"e_1_3_8_16_2","doi-asserted-by":"publisher","DOI":"10.1161\/01.CIR.0000158823.53941.BE"},{"key":"e_1_3_8_17_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-019-3173-2"},{"key":"e_1_3_8_18_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-015-0520-3"},{"key":"e_1_3_8_19_2","doi-asserted-by":"publisher","DOI":"10.1080\/03630242.2017.1421287"},{"key":"e_1_3_8_20_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2009.1242"},{"key":"e_1_3_8_21_2","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.15.00400"},{"key":"e_1_3_8_22_2","volume-title":"R: a language and environment for statistical computing","author":"R Core Team","year":"2020","unstructured":"R Core Team. 2020 R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. See https:\/\/www.r-project.org\/."},{"key":"e_1_3_8_23_2","unstructured":"Dragulescu A Arendt C. 2020 xlsx: read write format Excel 2007 and Excel 97\/2000\/XP\/2003 Files [Internet]. See https:\/\/cran.r-project.org\/package=xlsx."},{"key":"e_1_3_8_24_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01686"},{"key":"e_1_3_8_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24277-4"},{"key":"e_1_3_8_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2016.03.004"},{"key":"e_1_3_8_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2013.10.005"},{"key":"e_1_3_8_28_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1537592721000931"},{"key":"e_1_3_8_29_2","doi-asserted-by":"crossref","unstructured":"Heirene R et al. 2021 Preregistration specificity & adherence: a review of preregistered gambling studies & cross-disciplinary comparison. [cited 20 August 2021]; See https:\/\/psyarxiv.com\/nj4es\/.","DOI":"10.31234\/osf.io\/nj4es"},{"key":"e_1_3_8_30_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000937"},{"key":"e_1_3_8_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-018-0444-y"}],"container-title":"Royal Society Open Science","original-title":[],"language":"en","link":[{"URL":"https:\/\/royalsocietypublishing.org\/doi\/pdf\/10.1098\/rsos.211037","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/royalsocietypublishing.org\/doi\/full-xml\/10.1098\/rsos.211037","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/royalsocietypublishing.org\/doi\/pdf\/10.1098\/rsos.211037","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T07:05:50Z","timestamp":1635318350000},"score":1,"resource":{"primary":{"URL":"https:\/\/royalsocietypublishing.org\/doi\/10.1098\/rsos.211037"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10]]},"references-count":30,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2021,10]]}},"alternative-id":["10.1098\/rsos.211037"],"URL":"http:\/\/dx.doi.org\/10.1098\/rsos.211037","relation":{},"ISSN":["2054-5703"],"subject":["Multidisciplinary"],"container-title-short":"R. Soc. open sci.","published":{"date-parts":[[2021,10]]},"assertion":[{"value":"2021-06-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-29","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}' recorded_at: Wed, 05 Jul 2023 05:27:43 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.7554%2Felife.67995/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:43 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="application/pdf"; rel="item", ; version="vor"; type="application/xml"; rel="item", ; version="vor"; rel="license", ; version="am"; rel="license", ; version="tdm"; rel="license", ; title="Timothy M Errington"; rel="author", ; title="Nicole Perfito"; rel="author", ; title="Elizabeth Iorns"; rel="author", ; title="Brian A Nosek"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T18:17:31Z","timestamp":1687976251621},"reference-count":196,"publisher":"eLife Sciences Publications, Ltd","license":[{"start":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T00:00:00Z","timestamp":1638835200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T00:00:00Z","timestamp":1638835200000},"content-version":"am","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T00:00:00Z","timestamp":1638835200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100014848","name":"Arnold Ventures","doi-asserted-by":"crossref"}],"content-domain":{"domain":["elifesciences.org"],"crossmark-restriction":false},"accepted":{"date-parts":[[2021,7,20]]},"abstract":"We conducted the Reproducibility Project: Cancer Biology<\/jats:ext-link> to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely<\/jats:italic> or very helpful<\/jats:italic> for 41% of experiments, they were minimally helpful<\/jats:italic> for 9% of experiments, and not at all helpful<\/jats:italic> (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication \u2013 it is hard to assess whether reported findings are credible.<\/jats:p>","DOI":"10.7554\/elife.67995","type":"journal-article","created":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T00:00:20Z","timestamp":1638835220000},"update-policy":"http:\/\/dx.doi.org\/10.7554\/elife.67995","source":"Crossref","is-referenced-by-count":75,"title":"Challenges for assessing replicability in preclinical cancer biology","prefix":"10.7554","volume":"10","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-4959-5143","authenticated-orcid":true,"given":"Timothy M","family":"Errington","sequence":"first","affiliation":[{"name":"Center for Open Science","place":["Charlottesville, United States"]}]},{"given":"Alexandria","family":"Denis","sequence":"additional","affiliation":[{"name":"Center for Open Science","place":["Charlottesville, United States"]}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-9546-215X","authenticated-orcid":true,"given":"Nicole","family":"Perfito","sequence":"additional","affiliation":[{"name":"Science Exchange","place":["Palo Alto, United States"]}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5515-1258","authenticated-orcid":true,"given":"Elizabeth","family":"Iorns","sequence":"additional","affiliation":[{"name":"Science Exchange","place":["Palo Alto, United States"]}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-6797-5476","authenticated-orcid":true,"given":"Brian A","family":"Nosek","sequence":"additional","affiliation":[{"name":"Center for Open Science","place":["Charlottesville, United States"]},{"name":"University of Virginia","place":["Charlottesville, United States"]}]}],"member":"4374","published-online":{"date-parts":[[2021,12,7]]},"reference":[{"key":"bib1","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.21253","article-title":"Replication Study: BET bromodomain inhibition as a therapeutic strategy to target c-Myc","volume":"6","author":"Aird","year":"2017","journal-title":"eLife"},{"key":"bib2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000246","article-title":"Open science challenges, benefits and tips in early career and beyond","volume":"17","author":"Allen","year":"2019","journal-title":"PLOS Biology"},{"key":"bib3","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1525\/jer.2007.2.4.3","article-title":"Normative dissonance in science: results from a national survey of U.S. scientists","volume":"2","author":"Anderson","year":"2007","journal-title":"Journal of Empirical Research on Human Research Ethics"},{"key":"bib4","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1126\/science.1224820","article-title":"Intestinal inflammation targets cancer-inducing activity of the microbiota","volume":"338","author":"Arthur","year":"2012","journal-title":"Science"},{"key":"bib5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1001756","article-title":"Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies","volume":"12","author":"Baker","year":"2014","journal-title":"PLOS Biology"},{"key":"bib6","doi-asserted-by":"publisher","article-title":"Dutch agency launches first grants programme dedicated to replication","author":"Baker","year":"2016","journal-title":"Nature","DOI":"10.1038\/nature.2016.20287"},{"key":"bib7","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1038\/483531a","article-title":"Drug development: Raise standards for preclinical cancer research","volume":"483","author":"Begley","year":"2012","journal-title":"Nature"},{"key":"bib8","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1161\/CIRCRESAHA.114.303819","article-title":"Reproducibility in science: Improving the standard for basic and preclinical research","volume":"116","author":"Begley","year":"2015","journal-title":"Circulation Research"},{"key":"bib9","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1038\/nature11071","article-title":"Melanoma genome sequencing reveals frequent PREX2 mutations","volume":"485","author":"Berger","year":"2012","journal-title":"Nature"},{"key":"bib10","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.11999","article-title":"Registered Report: Kinase-dead BRAF and oncogenic RAS cooperate to drive tumor progression through CRAF","volume":"5","author":"Bhargava","year":"2016","journal-title":"eLife"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.09976","article-title":"Registered Report: RAF inhibitors prime wild-type RAF to activate the MAPK pathway and enhance growth","volume":"5","author":"Bhargava","year":"2016","journal-title":"eLife"},{"key":"bib12","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04034","article-title":"Registered Report: Tumour micro-environment elicits innate resistance to RAF inhibitors through HGF secretion","volume":"3","author":"Blum","year":"2014","journal-title":"eLife"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04024","article-title":"Registered Report: Transcriptional amplification in tumor cells with elevated c-Myc","volume":"4","author":"Blum","year":"2015","journal-title":"eLife"},{"key":"bib14","first-page":"437","article-title":"Effects of article retraction on citation and practice in medicine","volume":"87","author":"Budd","year":"1999","journal-title":"Bulletin of the Medical Library Association"},{"key":"bib15","doi-asserted-by":"publisher","first-page":"1433","DOI":"10.1126\/science.aaf0918","article-title":"Evaluating replicability of laboratory experiments in economics","volume":"351","author":"Camerer","year":"2016","journal-title":"Science"},{"key":"bib16","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1038\/s41562-018-0399-z","article-title":"Evaluating the replicability of social science experiments in nature and science between 2010 and 2015","volume":"2","author":"Camerer","year":"2018","journal-title":"Nature Human Behaviour"},{"key":"bib17","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1038\/nature08712","article-title":"The transcriptional network for mesenchymal transformation of brain tumours","volume":"463","author":"Carro","year":"2010","journal-title":"Nature"},{"key":"bib18","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1128\/IAI.06183-11","article-title":"Reforming science: methodological and cultural reforms","volume":"80","author":"Casadevall","year":"2012","journal-title":"Infection and Immunity"},{"key":"bib19","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1101\/gr.126516.111","article-title":"Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma","volume":"22","author":"Castellarin","year":"2012","journal-title":"Genome Research"},{"key":"bib20","doi-asserted-by":"publisher","first-page":"7950","DOI":"10.1073\/pnas.1102454108","article-title":"Normal and neoplastic nonstem cells can spontaneously convert to a stem-like state","volume":"108","author":"Chaffer","year":"2011","journal-title":"PNAS"},{"key":"bib21","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1016\/S0140-6736(13)62229-1","article-title":"How to increase value and reduce waste when research priorities are set","volume":"383","author":"Chalmers","year":"2014","journal-title":"Lancet"},{"key":"bib22","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1038\/d41586-019-02674-6","article-title":"What\u2019s next for registered reports","volume":"573","author":"Chambers","year":"2019","journal-title":"Nature"},{"key":"bib23","doi-asserted-by":"publisher","first-page":"522","DOI":"10.1038\/nature11287","article-title":"A restricted cell population propagates glioblastoma growth after chemotherapy","volume":"488","author":"Chen","year":"2012","journal-title":"Nature"},{"key":"bib24","doi-asserted-by":"publisher","article-title":"Open science practices are on the rise: The State of Social Science (3S) Survey","volume-title":"OSF Preprints","author":"Christensen","year":"2019","DOI":"10.31222\/osf.io\/5rksu"},{"key":"bib25","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/0749-5978(91)90010-Q","article-title":"The hindsight bias: A meta-analysis","volume":"48","author":"Christensen-Szalanski","year":"1991","journal-title":"Organizational Behavior and Human Decision Processes"},{"key":"bib26","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04180","article-title":"Registered Report: Melanoma genome sequencing reveals frequent PREX2 mutations","volume":"3","author":"Chroscinski","year":"2014","journal-title":"eLife"},{"key":"bib27","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04586","article-title":"Registered Report: The CD47-signal regulated protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors","volume":"4","author":"Chroscinski","year":"2015","journal-title":"eLife"},{"key":"bib28","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04363","article-title":"Registered Report: Tumour vascularization via endothelial differentiation of glioblastoma stem-like cells","volume":"4","author":"Chroscinski","year":"2015","journal-title":"eLife"},{"key":"bib29","article-title":"Dear colleague letter: Robust and reliable research in the social, behavioral, and economic sciences","volume-title":"National Science Foundation","author":"Cook","year":"2016","unstructured":"Cook FL. 2016. Dear colleague letter: Robust and reliable research in the social, behavioral, and economic sciences. National Science Foundation. https:\/\/www.nsf.gov\/pubs\/2016\/nsf16137\/nsf16137.jsp [Accessed August 5, 2021]."},{"key":"bib30","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s13164-018-0400-9","article-title":"Estimating the reproducibility of experimental philosophy","volume":"12","author":"Cova","year":"2018","journal-title":"Review of Philosophy and Psychology"},{"key":"bib31","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1038\/nature10802","article-title":"DNA breaks and chromosome pulverization from errors in mitosis","volume":"482","author":"Crasta","year":"2012","journal-title":"Nature"},{"key":"bib32","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature10509","article-title":"Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukaemia","volume":"478","author":"Dawson","year":"2011","journal-title":"Nature"},{"key":"bib33","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1016\/j.cell.2011.08.017","article-title":"BET bromodomain inhibition as a therapeutic strategy to target c-Myc","volume":"146","author":"Delmore","year":"2011","journal-title":"Cell"},{"key":"bib34","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1038\/nature10189","article-title":"Oncogene-induced Nrf2 transcription promotes ROS detoxification and tumorigenesis","volume":"475","author":"DeNicola","year":"2011","journal-title":"Nature"},{"key":"bib35","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1038\/nature11344","article-title":"Defining the mode of tumour growth by clonal analysis","volume":"488","author":"Driessens","year":"2012","journal-title":"Nature"},{"key":"bib36","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1016\/j.cmet.2016.08.006","article-title":"Never waste a good crisis: Confronting reproducibility in translational research","volume":"24","author":"Drucker","year":"2016","journal-title":"Cell Metabolism"},{"key":"bib37","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04186","article-title":"Registered Report: Intestinal inflammation targets cancer-inducing activity of the microbiota","volume":"4","author":"Eaton","year":"2015","journal-title":"eLife"},{"key":"bib38","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.34364","article-title":"Replication Study: Intestinal inflammation targets cancer-inducing activity of the microbiota","volume":"7","author":"Eaton","year":"2018","journal-title":"eLife"},{"key":"bib39","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1016\/j.jesp.2015.10.012","article-title":"Many Labs 3: Evaluating participant pool quality across the academic semester via replication","volume":"67","author":"Ebersole","year":"2016","journal-title":"Journal of Experimental Social Psychology"},{"key":"bib40","doi-asserted-by":"publisher","article-title":"Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability","volume-title":"PsyArXiv","author":"Ebersole","year":"2019","DOI":"10.31234\/osf.io\/sxfm2"},{"key":"bib41","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04333","article-title":"An open investigation of the reproducibility of cancer biology research","volume":"3","author":"Errington","year":"2014","journal-title":"eLife"},{"key":"bib42","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.73430","article-title":"Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology","volume":"10","author":"Errington","year":"2021","journal-title":"eLife"},{"key":"bib43","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.71601","article-title":"Investigating the replicability of preclinical cancer biology","volume":"10","author":"Errington","year":"2021","journal-title":"eLife"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.45426","article-title":"Replication Study: Wnt activity defines colon cancer stem cells and is regulated by the microenvironment","volume":"8","author":"Essex","year":"2019","journal-title":"eLife"},{"key":"bib45","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.07301","article-title":"Registered report: Wnt activity defines colon cancer stem cells and is regulated by the microenvironment","volume":"4","author":"Evans","year":"2015","journal-title":"eLife"},{"key":"bib46","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.07420","article-title":"Registered Report: Oncometabolite 2-hydroxyglutarate is a competitive inhibitor of \u03b1-ketoglutarate-dependent dioxygenases","volume":"4","author":"Evans","year":"2015","journal-title":"eLife"},{"key":"bib47","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0010068","article-title":"\u201cPositive\u201d results increase down the hierarchy of the sciences","volume":"5","author":"Fanelli","year":"2010","journal-title":"PLOS ONE"},{"key":"bib48","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1007\/s11192-011-0494-7","article-title":"Negative results are disappearing from most disciplines and countries","volume":"90","author":"Fanelli","year":"2011","journal-title":"Scientometrics"},{"key":"bib49","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.12626","article-title":"Registered Report: The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate","volume":"5","author":"Fiehn","year":"2016","journal-title":"eLife"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04796","article-title":"Registered Report: Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis","volume":"4","author":"Fiering","year":"2015","journal-title":"eLife"},{"key":"bib51","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1016\/j.ccr.2010.11.015","article-title":"Leukemic IDH1 and IDH2 mutations result in a hypermethylation phenotype, disrupt TET2 function, and impair hematopoietic differentiation","volume":"18","author":"Figueroa","year":"2010","journal-title":"Cancer Cell"},{"key":"bib52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0030-5073(75)90002-1","article-title":"I knew it would happen: Remembered probabilities of once\u2014future things","volume":"13","author":"Fischhoff","year":"1975","journal-title":"Organizational Behavior and Human Performance"},{"key":"bib53","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.08997","article-title":"Registered Report: Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukemia","volume":"4","author":"Fung","year":"2015","journal-title":"eLife"},{"key":"bib54","doi-asserted-by":"publisher","first-page":"570","DOI":"10.1038\/nature11005","article-title":"Systematic identification of genomic markers of drug sensitivity in cancer cells","volume":"483","author":"Garnett","year":"2012","journal-title":"Nature"},{"key":"bib55","article-title":"The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No \u201cFishing Expedition\u201d or \u201cp-Hacking\u201d and the Research Hypothesis Was Posited Ahead of Time","volume-title":"Department of Statistics, Columbia University","author":"Gelman","year":"2013","unstructured":"Gelman A, Loken E. 2013. The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No \u201cFishing Expedition\u201d or \u201cp-Hacking\u201d and the Research Hypothesis Was Posited Ahead of Time. Department of Statistics, Columbia University. http:\/\/www.stat.columbia.edu\/~gelman\/research\/unpublished\/forking.pdf [Accessed August 5, 2021]."},{"key":"bib56","doi-asserted-by":"publisher","first-page":"562","DOI":"10.1177\/1745691612457576","article-title":"Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science","volume":"7","author":"Giner-Sorolla","year":"2012","journal-title":"Perspectives on Psychological Science"},{"key":"bib57","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1016\/S0140-6736(13)62228-X","article-title":"Reducing waste from incomplete or unusable reports of biomedical research","volume":"383","author":"Glasziou","year":"2014","journal-title":"Lancet"},{"key":"bib58","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1016\/j.cell.2011.05.040","article-title":"Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis","volume":"146","author":"Goetz","year":"2011","journal-title":"Cell"},{"key":"bib59","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04037","article-title":"Registered Report: Widespread potential for growth factor-driven resistance to anticancer kinase inhibitors","volume":"3","author":"Greenfield","year":"2014","journal-title":"eLife"},{"key":"bib60","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1037\/h0076157","article-title":"Consequences of prejudice against the null hypothesis","volume":"82","author":"Greenwald","year":"1975","journal-title":"Psychological Bulletin"},{"key":"bib61","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pntd.0004194","article-title":"Quality of reporting and adherence to arrive guidelines in animal studies for chagas disease preclinical drug research: A systematic review","volume":"9","author":"Gulin","year":"2015","journal-title":"PLOS Neglected Tropical Diseases"},{"key":"bib62","doi-asserted-by":"publisher","first-page":"1071","DOI":"10.1038\/nature08975","article-title":"Long non-coding RNA HOTAIR reprograms chromatin state to promote cancer metastasis","volume":"464","author":"Gupta","year":"2010","journal-title":"Nature"},{"key":"bib63","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0183591","article-title":"A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review","volume":"12","author":"Han","year":"2017","journal-title":"PLOS ONE"},{"key":"bib64","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1037\/a0015701","article-title":"Feeling validated versus being correct: A meta-analysis of selective exposure to information","volume":"135","author":"Hart","year":"2009","journal-title":"Psychological Bulletin"},{"key":"bib65","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1038\/nature08833","article-title":"RAF inhibitors prime wild-type RAF to activate the MAPK pathway and enhance growth","volume":"464","author":"Hatzivassiliou","year":"2010","journal-title":"Nature"},{"key":"bib66","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.09462","article-title":"Registered Report: A chromatin-mediated reversible drug-tolerant state in cancer cell subpopulations","volume":"5","author":"Haven","year":"2016","journal-title":"eLife"},{"key":"bib67","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1016\/j.cell.2009.12.040","article-title":"Kinase-dead BRAF and oncogenic RAS cooperate to drive tumor progression through CRAF","volume":"140","author":"Heidorn","year":"2010","journal-title":"Cell"},{"key":"bib68","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.18173","article-title":"Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors","volume":"6","author":"Horrigan","year":"2017","journal-title":"eLife"},{"key":"bib69","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.21634","article-title":"Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations","volume":"6","author":"Horrigan","year":"2017","journal-title":"eLife"},{"key":"bib70","doi-asserted-by":"publisher","article-title":"Evaluating Registered Reports: A Naturalistic Comparative Study of Article Impact","volume-title":"OSF","author":"Hummer","year":"2017","DOI":"10.31219\/osf.io\/5y8w7"},{"key":"bib71","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.06938","article-title":"Registered Report: Interactions between cancer stem cells and their niche govern metastatic colonization","volume":"4","author":"Incardona","year":"2015","journal-title":"eLife"},{"key":"bib72","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"Ioannidis","year":"2005","journal-title":"PLOS Medicine"},{"key":"bib73","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1016\/S0140-6736(13)62227-8","article-title":"Increasing value and reducing waste in research design, conduct, and analysis","volume":"383","author":"Ioannidis","year":"2014","journal-title":"Lancet"},{"key":"bib74","doi-asserted-by":"publisher","first-page":"968","DOI":"10.1038\/nature09627","article-title":"COT drives resistance to RAF inhibition through MAP kinase pathway reactivation","volume":"468","author":"Johannessen","year":"2010","journal-title":"Nature"},{"key":"bib75","doi-asserted-by":"publisher","first-page":"524","DOI":"10.1177\/0956797611430953","article-title":"Measuring the prevalence of questionable research practices with incentives for truth telling","volume":"23","author":"John","year":"2012","journal-title":"Psychological Science"},{"key":"bib76","doi-asserted-by":"publisher","first-page":"869","DOI":"10.1038\/nature09208","article-title":"Diverse somatic mutation patterns and pathway alterations in human cancers","volume":"466","author":"Kan","year":"2010","journal-title":"Nature"},{"key":"bib77","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.06959","article-title":"Registered Report: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs","volume":"4","author":"Kandela","year":"2015","journal-title":"eLife"},{"key":"bib78","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.07072","article-title":"Registered Report: BET bromodomain inhibition as a therapeutic strategy to target c-Myc","volume":"4","author":"Kandela","year":"2015","journal-title":"eLife"},{"key":"bib79","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.06847","article-title":"Registered Report: Discovery and preclinical validation of drug indications using compendia of public gene expression data","volume":"4","author":"Kandela","year":"2015","journal-title":"eLife"},{"key":"bib80","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.17044","article-title":"Replication Study: Discovery and preclinical validation of drug indications using compendia of public gene expression data","volume":"6","author":"Kandela","year":"2017","journal-title":"eLife"},{"key":"bib81","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1038\/nature10599","article-title":"Senescence surveillance of pre-malignant hepatocytes limits liver cancer development","volume":"479","author":"Kang","year":"2011","journal-title":"Nature"},{"key":"bib82","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0132382","article-title":"Likelihood of null effects of large NHLBI clinical trials has Increased over time","volume":"10","author":"Kaplan","year":"2015","journal-title":"PLOS ONE"},{"key":"bib83","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1207\/s15327957pspr0203_4","article-title":"HARKing: hypothesizing after the results are known","volume":"2","author":"Kerr","year":"1998","journal-title":"Personality and Social Psychology Review"},{"key":"bib84","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.51019","article-title":"Replication Study: A coding-independent function of gene and pseudogene mRNAs regulates tumour biology","volume":"9","author":"Kerwin","year":"2020","journal-title":"eLife"},{"key":"bib85","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.08245","article-title":"Registered Report: A coding-independent function of gene and pseudogene mRNAs regulates tumour biology","volume":"4","author":"Khan","year":"2015","journal-title":"eLife"},{"key":"bib86","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0007824","article-title":"Survey of the quality of experimental design, statistical analysis and reporting of research using animals","volume":"4","author":"Kilkenny","year":"2009","journal-title":"PLOS ONE"},{"key":"bib87","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.39944","article-title":"Replication study: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET","volume":"7","author":"Kim","year":"2018","journal-title":"eLife"},{"key":"bib88","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1027\/1864-9335\/a000178","article-title":"Investigating variation in replicability: A \u201cMany Labs\u201d replication project","volume":"45","author":"Klein","year":"2014","journal-title":"Social Psychology"},{"key":"bib89","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1177\/2515245918810225","article-title":"Many Labs 2: Investigating variation in replicability across samples and settings","volume":"1","author":"Klein","year":"2018","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"bib90","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1038\/nature09586","article-title":"Impaired hydroxylation of 5-methylcytosine in myeloid cancers with mutant TET2","volume":"468","author":"Ko","year":"2010","journal-title":"Nature"},{"key":"bib91","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1037\/0033-2909.108.3.480","article-title":"The case for motivated reasoning","volume":"108","author":"Kunda","year":"1990","journal-title":"Psychological Bulletin"},{"key":"bib92","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1038\/nature11556","article-title":"A call for transparent reporting to optimize the predictive value of preclinical research","volume":"490","author":"Landis","year":"2012","journal-title":"Nature"},{"key":"bib93","doi-asserted-by":"publisher","first-page":"780","DOI":"10.1016\/j.cell.2012.03.031","article-title":"Sequential application of anticancer drugs enhances cell death by rewiring apoptotic signaling networks","volume":"149","author":"Lee","year":"2012","journal-title":"Cell"},{"key":"bib94","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.07383","article-title":"Registered Report: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET","volume":"5","author":"Lesnik","year":"2016","journal-title":"eLife"},{"key":"bib95","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.30274","article-title":"Replication Study: Transcriptional amplification in tumor cells with elevated c-Myc","volume":"7","author":"Lewis","year":"2018","journal-title":"eLife"},{"key":"bib96","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.06434","article-title":"Registered Report: The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44","volume":"4","author":"Li","year":"2015","journal-title":"eLife"},{"key":"bib97","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.cell.2012.08.026","article-title":"Transcriptional amplification in tumor cells with elevated c-Myc","volume":"151","author":"Lin","year":"2012","journal-title":"Cell"},{"key":"bib98","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1038\/nm.2284","article-title":"The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44","volume":"17","author":"Liu","year":"2011","journal-title":"Nature Medicine"},{"key":"bib99","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1038\/522151a","article-title":"Reproducibility: Use mouse biobanks or lose them","volume":"522","author":"Lloyd","year":"2015","journal-title":"Nature"},{"key":"bib100","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1038\/nature10860","article-title":"IDH mutation impairs histone demethylation and results in a block to cell differentiation","volume":"483","author":"Lu","year":"2012","journal-title":"Nature"},{"key":"bib101","doi-asserted-by":"publisher","DOI":"10.1038\/srep03146","article-title":"The retraction penalty: Evidence from the Web of Science","volume":"3","author":"Lu","year":"2013","journal-title":"Scientific Reports"},{"key":"bib102","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1016\/S0140-6736(13)62329-6","article-title":"Biomedical research: Increasing value, reducing waste","volume":"383","author":"Macleod","year":"2014","journal-title":"Lancet"},{"key":"bib103","doi-asserted-by":"publisher","article-title":"Findings of a Retrospective, Controlled Cohort Study of the Impact of a Change in Nature Journals\u2019 Editorial Policy for Life Sciences Research on the Completeness of Reporting Study Design and Execution","volume-title":"bioRxiv","author":"Macleod","year":"2017","DOI":"10.1101\/187245"},{"key":"bib104","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2103238118","article-title":"The MDAR (Materials Design Analysis Reporting) Framework for transparent reporting in the life sciences","volume":"118","author":"Macleod","year":"2021","journal-title":"PNAS"},{"key":"bib105","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s11948-014-9532-1","article-title":"The (lack of) impact of retraction on citation networks","volume":"21","author":"Madlock-Brown","year":"2015","journal-title":"Science and Engineering Ethics"},{"key":"bib106","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/BF01173636","article-title":"Publication prejudices: An experimental study of confirmatory bias in the peer review system","volume":"1","author":"Mahoney","year":"1977","journal-title":"Cognitive Therapy and Research"},{"key":"bib107","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1177\/1745691612460688","article-title":"Replications in psychology research: How often do they really occur?","volume":"7","author":"Makel","year":"2012","journal-title":"Perspectives on Psychological Science"},{"key":"bib108","doi-asserted-by":"publisher","first-page":"304","DOI":"10.3102\/0013189X14545513","article-title":"Facts are more important than novelty: Replication in the education sciences","volume":"43","author":"Makel","year":"2014","journal-title":"Educational Researcher"},{"key":"bib109","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1038\/nature10694","article-title":"Interactions between cancer stem cells and their niche govern metastatic colonization","volume":"481","author":"Malanchi","year":"2011","journal-title":"Nature"},{"key":"bib110","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.17584","article-title":"Replication Study: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs","volume":"6","author":"Mantis","year":"2017","journal-title":"eLife"},{"key":"bib111","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1016\/j.cell.2016.08.021","article-title":"A STAR Is born","volume":"166","author":"Marcus","year":"2016","journal-title":"Cell"},{"key":"bib112","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1080\/08109029208629515","article-title":"Scientific fraud and the power structure of science","volume":"10","author":"Martin","year":"1992","journal-title":"Prometheus"},{"key":"bib113","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.16800","article-title":"How open science helps researchers succeed","volume":"5","author":"McKiernan","year":"2016","journal-title":"eLife"},{"key":"bib114","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1126\/science.aad7048","article-title":"Liberating field science samples and data","volume":"351","author":"McNutt","year":"2016","journal-title":"Science"},{"key":"bib115","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1038\/nature10602","article-title":"Reductive glutamine metabolism by IDH1 mediates lipogenesis under hypoxia","volume":"481","author":"Metallo","year":"2011","journal-title":"Nature"},{"key":"bib116","doi-asserted-by":"publisher","DOI":"10.1186\/1741-7015-6-13","article-title":"Helping editors, peer reviewers and authors improve the clarity, completeness and transparency of reporting health research","volume":"6","author":"Moher","year":"2008","journal-title":"BMC Medicine"},{"key":"bib117","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1001195","article-title":"The Open Knowledge Foundation: open data means better science","volume":"9","author":"Molloy","year":"2011","journal-title":"PLOS Biology"},{"key":"bib118","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1038\/ng.518","article-title":"Somatic mutations altering EZH2 (Tyr641) in follicular and diffuse large B-cell lymphomas of germinal-center origin","volume":"42","author":"Morin","year":"2010","journal-title":"Nature Genetics"},{"key":"bib119","article-title":"Panton Principles, Principles for Open Data in Science","author":"Murray-Rust","year":"2010","unstructured":"Murray-Rust P, Neylon C, Pollock R, Wilbanks J. 2010. Panton Principles, Principles for Open Data in Science. http:\/\/pantonprinciples.org [Accessed August 5, 2021]."},{"key":"bib120","doi-asserted-by":"publisher","volume-title":"Reproducibility and Replicability in Science","author":"NAS","year":"2019","DOI":"10.17226\/25303"},{"key":"bib121","doi-asserted-by":"publisher","DOI":"10.1038\/496398a","article-title":"Announcement: Reducing our irreproducibility","volume":"496","author":"Nature","year":"2013","journal-title":"Nature"},{"key":"bib122","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1038\/nature09626","article-title":"Melanomas acquire resistance to B-RAF(V600E) inhibition by RTK or N-RAS upregulation","volume":"468","author":"Nazarian","year":"2010","journal-title":"Nature"},{"key":"bib123","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1037\/1089-2680.2.2.175","article-title":"Confirmation bias: A ubiquitous phenomenon in many guises","volume":"2","author":"Nickerson","year":"1998","journal-title":"Review of General Psychology"},{"key":"bib124","doi-asserted-by":"publisher","first-page":"615","DOI":"10.1177\/1745691612459058","article-title":"Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability","volume":"7","author":"Nosek","year":"2012","journal-title":"Perspectives on Psychological Science"},{"key":"bib125","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1027\/1864-9335\/a000192","article-title":"Registered Reports: A method to increase the credibility of published results","volume":"45","author":"Nosek","year":"2014","journal-title":"Social Psychology"},{"key":"bib126","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1126\/science.aab2374","article-title":"Promoting an open research culture","volume":"348","author":"Nosek","year":"2015","journal-title":"Science"},{"key":"bib127","doi-asserted-by":"publisher","first-page":"2600","DOI":"10.1073\/pnas.1708274114","article-title":"The preregistration revolution","volume":"115","author":"Nosek","year":"2018","journal-title":"PNAS"},{"key":"bib128","article-title":"Preregistration Becoming the Norm in Psychological Science","author":"Nosek","year":"2018","unstructured":"Nosek BA, Lindsay DS. 2018. Preregistration Becoming the Norm in Psychological Science. https:\/\/www.psychologicalscience.org\/observer\/preregistration-becoming-the-norm-in-psychological-science [Accessed August 5, 2021]."},{"key":"bib129","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1016\/j.tics.2019.07.009","article-title":"Preregistration Is hard, and worthwhile","volume":"23","author":"Nosek","year":"2019","journal-title":"Trends in Cognitive Sciences"},{"key":"bib130","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1038\/d41586-020-02142-6","article-title":"The best time to argue about what a replication means? Before you do it","volume":"583","author":"Nosek","year":"2020","journal-title":"Nature"},{"key":"bib131","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000691","article-title":"What is replication?","volume":"18","author":"Nosek","year":"2020","journal-title":"PLOS Biology"},{"key":"bib132","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac4716","article-title":"Psychology Estimating the reproducibility of psychological science","volume":"349","author":"Open Science Collaboration","year":"2015","journal-title":"Science"},{"key":"bib133","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1038\/nature10491","article-title":"An endogenous tumour-promoting ligand of the human aryl hydrocarbon receptor","volume":"478","author":"Opitz","year":"2011","journal-title":"Nature"},{"key":"bib134","doi-asserted-by":"publisher","first-page":"883","DOI":"10.1038\/nm.2753","article-title":"Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET","volume":"18","author":"Peinado","year":"2012","journal-title":"Nature Medicine"},{"key":"bib135","doi-asserted-by":"publisher","article-title":"Replication Study: RAF Inhibitors Prime Wild-Type RAF to Activate the MAPK Pathway and Enhance Growth","volume-title":"bioRxiv","author":"Pelech","year":"2021","DOI":"10.1101\/2021.11.30.470372"},{"key":"bib136","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1038\/507423a","article-title":"Preclinical research: Make mouse studies work","volume":"507","author":"Perrin","year":"2014","journal-title":"Nature"},{"key":"bib137","doi-asserted-by":"publisher","first-page":"1420","DOI":"10.1001\/jama.1990.03440100140020","article-title":"The continued use of retracted, invalid scientific literature","volume":"263","author":"Pfeifer","year":"1990","journal-title":"JAMA"},{"key":"bib138","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.12470","article-title":"Registered Report: Coding-independent regulation of the tumor suppressor PTEN by competing endogenous mRNAs","volume":"5","author":"Phelps","year":"2016","journal-title":"eLife"},{"key":"bib139","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1038\/nature10661","article-title":"A microRNA regulon that mediates endothelial recruitment and metastasis by cancer cells","volume":"481","author":"Png","year":"2011","journal-title":"Nature"},{"key":"bib140","doi-asserted-by":"publisher","first-page":"1033","DOI":"10.1038\/nature09144","article-title":"A coding-independent function of gene and pseudogene mRNAs regulates tumour biology","volume":"465","author":"Poliseno","year":"2010","journal-title":"Nature"},{"key":"bib141","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1038\/nature10350","article-title":"Functional genomics reveal that the serine synthesis pathway is essential in breast cancer","volume":"476","author":"Possemato","year":"2011","journal-title":"Nature"},{"key":"bib142","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1038\/nature08902","article-title":"RAF inhibitors transactivate RAF dimers and ERK signalling in cells with wild-type BRAF","volume":"464","author":"Poulikakos","year":"2010","journal-title":"Nature"},{"key":"bib143","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1038\/nature10868","article-title":"Unresponsiveness of colon cancer to BRAF(V600E) inhibition through feedback activation of EGFR","volume":"483","author":"Prahallad","year":"2012","journal-title":"Nature"},{"key":"bib144","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1146\/annurev-criminol-032317-091849","article-title":"Replication in criminology and the social sciences","volume":"1","author":"Pridemore","year":"2018","journal-title":"Annual Review of Criminology"},{"key":"bib145","doi-asserted-by":"publisher","DOI":"10.1038\/nrd3439-c1","article-title":"Believe it or not: How much can we rely on published data on potential drug targets","volume":"10","author":"Prinz","year":"2011","journal-title":"Nature Reviews Drug Discovery"},{"key":"bib146","volume-title":"Philosophical Papers: Mathematics, Matter, and Method","author":"Putnam","year":"1975"},{"key":"bib147","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1038\/nature10138","article-title":"CCL2 recruits inflammatory monocytes to facilitate breast-tumour metastasis","volume":"475","author":"Qian","year":"2011","journal-title":"Nature"},{"key":"bib148","article-title":"R: A Language and Environment for Statistical Computing","author":"R Development Core Team","year":"2021","unstructured":"R Development Core Team. 2021. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https:\/\/www.R-project.org\/."},{"key":"bib149","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1038\/nature10167","article-title":"Selective killing of cancer cells by a small molecule targeting the stress response to ROS","volume":"475","author":"Raj","year":"2011","journal-title":"Nature"},{"key":"bib150","doi-asserted-by":"publisher","first-page":"1916","DOI":"10.1161\/CIRCRESAHA.117.310628","article-title":"Methodological rigor in preclinical cardiovascular studies","volume":"120","author":"Ramirez","year":"2017","journal-title":"Circulation Research"},{"key":"bib151","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.04105","article-title":"Registered Report: Senescence surveillance of pre-malignant hepatocytes limits liver cancer development","volume":"4","author":"Raouf","year":"2015","journal-title":"eLife"},{"key":"bib152","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1038\/d41586-020-00672-7","article-title":"A controlled trial for reproducibility","volume":"579","author":"Raphael","year":"2020","journal-title":"Nature"},{"key":"bib153","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.10012","article-title":"Registered Report: Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma","volume":"5","author":"Repass","year":"2016","journal-title":"eLife"},{"key":"bib154","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.25801","article-title":"Replication Study: Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma","volume":"7","author":"Repass","year":"2018","journal-title":"eLife"},{"key":"bib155","doi-asserted-by":"publisher","first-page":"824","DOI":"10.1038\/nature09557","article-title":"Tumour vascularization via endothelial differentiation of glioblastoma stem-like cells","volume":"468","author":"Ricci-Vitiani","year":"2010","journal-title":"Nature"},{"key":"bib156","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.10860","article-title":"Registered Report: IDH mutation impairs histone demethylation and results in a block to cell differentiation","volume":"5","author":"Richarson","year":"2016","journal-title":"eLife"},{"key":"bib157","doi-asserted-by":"publisher","first-page":"638","DOI":"10.1037\/0033-2909.86.3.638","article-title":"The file drawer problem and tolerance for null results","volume":"86","author":"Rosenthal","year":"1979","journal-title":"Psychological Bulletin"},{"key":"bib158","doi-asserted-by":"publisher","DOI":"10.1177\/25152459211007467","article-title":"An excess of positive results: Comparing the standard psychology literature with Registered Reports","volume":"4","author":"Scheel","year":"2020","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"bib159","doi-asserted-by":"publisher","first-page":"730","DOI":"10.1126\/science.1224676","article-title":"Lineage tracing reveals Lgr5+ stem cell activity in mouse intestinal adenomas","volume":"337","author":"Schepers","year":"2012","journal-title":"Science"},{"key":"bib160","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1037\/a0015108","article-title":"Shall we really do it again? The powerful concept of replication is neglected in the social sciences","volume":"13","author":"Schmidt","year":"2009","journal-title":"Review of General Psychology"},{"key":"bib161","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3001107","article-title":"Assessment of transparency indicators across the biomedical literature: How open is open","volume":"19","author":"Serghiou","year":"2021","journal-title":"PLOS Biology"},{"key":"bib162","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.25306","article-title":"Replication Study: Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukaemia","volume":"6","author":"Shan","year":"2017","journal-title":"eLife"},{"key":"bib163","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.cell.2010.02.027","article-title":"A chromatin-mediated reversible drug-tolerant state in cancer cell subpopulations","volume":"141","author":"Sharma","year":"2010","journal-title":"Cell"},{"key":"bib164","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.11566","article-title":"Registered Report: Diverse somatic mutation patterns and pathway alterations in human cancers","volume":"5","author":"Sharma","year":"2016","journal-title":"eLife"},{"key":"bib165","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.11414","article-title":"Registered Report: COT drives resistance to RAF inhibition through MAP kinase pathway reactivation","volume":"5","author":"Sharma","year":"2016","journal-title":"eLife"},{"key":"bib166","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.45120","article-title":"Replication Study: Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis","volume":"8","author":"Sheen","year":"2019","journal-title":"eLife"},{"key":"bib167","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.26030","article-title":"Replication Study: The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate","volume":"6","author":"Showalter","year":"2017","journal-title":"eLife"},{"key":"bib168","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"Simmons","year":"2011","journal-title":"Psychological Science"},{"key":"bib169","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.3001318","article-title":"Discovery and preclinical validation of drug indications using compendia of public gene expression data","volume":"3","author":"Sirota","year":"2011","journal-title":"Science Translational Medicine"},{"key":"bib170","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.160384","article-title":"The natural selection of bad science","volume":"3","author":"Smaldino","year":"2016","journal-title":"Royal Society Open Science"},{"key":"bib171","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1038\/s41562-021-01142-4","article-title":"Initial evidence of research quality of registered reports compared with the standard publishing model","volume":"5","author":"Soderberg","year":"2021","journal-title":"Nature Human Behaviour"},{"key":"bib172","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1007\/s11673-008-9113-6","article-title":"Exploring scientific misconduct: Isolated individuals, impure institutions, or an inevitable idiom of modern science","volume":"5","author":"Sovacool","year":"2008","journal-title":"Journal of Bioethical Inquiry"},{"key":"bib173","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1080\/01621459.1959.10501497","article-title":"Publication decisions and their possible effects on inferences drawn from tests of significance\u2014or vice versa","volume":"54","author":"Sterling","year":"1959","journal-title":"Journal of the American Statistical Association"},{"key":"bib174","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1080\/00031305.1995.10476125","article-title":"Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa","volume":"49","author":"Sterling","year":"1995","journal-title":"The American Statistician"},{"key":"bib175","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1016\/j.expneurol.2011.06.017","article-title":"Replication and reproducibility in spinal cord injury research","volume":"233","author":"Steward","year":"2012","journal-title":"Experimental Neurology"},{"key":"bib176","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0067111","article-title":"Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals","volume":"8","author":"Stodden","year":"2013","journal-title":"PLOS ONE"},{"key":"bib177","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1038\/nature11183","article-title":"Tumour micro-environment elicits innate resistance to RAF inhibitors through HGF secretion","volume":"487","author":"Straussman","year":"2012","journal-title":"Nature"},{"key":"bib178","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.1126\/science.1183057","article-title":"Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs","volume":"328","author":"Sugahara","year":"2010","journal-title":"Science"},{"key":"bib179","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.cell.2011.09.041","article-title":"An extensive microRNA-mediated network of RNA-RNA interactions regulates established oncogenic pathways in glioblastoma","volume":"147","author":"Sumazin","year":"2011","journal-title":"Cell"},{"key":"bib180","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1016\/j.cell.2011.09.029","article-title":"Coding-independent regulation of the tumor suppressor PTEN by competing endogenous mRNAs","volume":"147","author":"Tay","year":"2011","journal-title":"Cell"},{"key":"bib181","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s11121-011-0217-6","article-title":"Replication in prevention science","volume":"12","author":"Valentine","year":"2011","journal-title":"Prevention Science"},{"key":"bib182","doi-asserted-by":"publisher","DOI":"10.1136\/bmjos-2019-100051","article-title":"Publication rate in preclinical research: a plea for preregistration","volume":"4","author":"van der Naald","year":"2020","journal-title":"BMJ Open Science"},{"key":"bib183","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.13620","article-title":"Registered Report: Systematic identification of genomic markers of drug sensitivity in cancer cells","volume":"5","author":"Vanden Heuvel","year":"2016","journal-title":"eLife"},{"key":"bib184","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.29747","article-title":"Replication Study: Systematic identification of genomic markers of drug sensitivity in cancer cells","volume":"7","author":"Vanden Heuvel","year":"2018","journal-title":"eLife"},{"key":"bib185","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1038\/ncb2048","article-title":"Wnt activity defines colon cancer stem cells and is regulated by the microenvironment","volume":"12","author":"Vermeulen","year":"2010","journal-title":"Nature Cell Biology"},{"key":"bib186","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1177\/1745691612463078","article-title":"An agenda for purely confirmatory research","volume":"7","author":"Wagenmakers","year":"2012","journal-title":"Perspectives on Psychological Science"},{"key":"bib187","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.56651","article-title":"Replication Study: Coding-independent regulation of the tumor suppressor PTEN by competing endogenous mRNAs","volume":"9","author":"Wang","year":"2020","journal-title":"eLife"},{"key":"bib188","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1016\/j.ccr.2010.01.020","article-title":"The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate","volume":"17","author":"Ward","year":"2010","journal-title":"Cancer Cell"},{"key":"bib189","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18","article-title":"The FAIR Guiding Principles for scientific data management and stewardship","volume":"3","author":"Wilkinson","year":"2016","journal-title":"Scientific Data"},{"key":"bib190","doi-asserted-by":"publisher","first-page":"6662","DOI":"10.1073\/pnas.1121623109","article-title":"The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors","volume":"109","author":"Willingham","year":"2012","journal-title":"PNAS"},{"key":"bib191","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1038\/nature11249","article-title":"Widespread potential for growth-factor-driven resistance to anticancer kinase inhibitors","volume":"487","author":"Wilson","year":"2012","journal-title":"Nature"},{"key":"bib192","article-title":"ACD Working Group on Enhancing Rigor, Transparency, and Translatability in Animal Research","author":"Wold","year":"2021","unstructured":"Wold B, Tabak LA, Ator N, Berro L, Bliss-Moreau E, Gonzalez-Villalobos RA, Hankenson C, Kiermer V, Mathis KW, Nusser S, Nuzzo R, Prager E, Ramirez FD, Svenson K, Berridge B, Brown P, Clayton J, Gordon JA, Lauer M, Wolinetz C. 2021. ACD Working Group on Enhancing Rigor, Transparency, and Translatability in Animal Research. https:\/\/www.acd.od.nih.gov\/documents\/presentations\/06112021_ACD_WorkingGroup_FinalReport.pdf [Accessed August 5, 2021]."},{"key":"bib193","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.ccr.2010.12.014","article-title":"Oncometabolite 2-hydroxyglutarate is a competitive inhibitor of \u03b1-ketoglutarate-dependent dioxygenases","volume":"19","author":"Xu","year":"2011","journal-title":"Cancer Cell"},{"key":"bib194","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.43511","article-title":"Replication Study: The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44","volume":"8","author":"Yan","year":"2019","journal-title":"eLife"},{"key":"bib195","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1038\/nature10371","article-title":"BRCA1 tumour suppression occurs via heterochromatin-mediated silencing","volume":"477","author":"Zhu","year":"2011","journal-title":"Nature"},{"key":"bib196","doi-asserted-by":"publisher","first-page":"524","DOI":"10.1038\/nature10334","article-title":"RNAi screen identifies Brd4 as a therapeutic target in acute myeloid leukaemia","volume":"478","author":"Zuber","year":"2011","journal-title":"Nature"}],"container-title":"eLife","original-title":[],"language":"en","link":[{"URL":"https:\/\/cdn.elifesciences.org\/articles\/67995\/elife-67995-v2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/cdn.elifesciences.org\/articles\/67995\/elife-67995-v2.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"}],"deposited":{"date-parts":[[2023,6,14]],"date-time":"2023-06-14T14:55:21Z","timestamp":1686754521000},"score":1,"resource":{"primary":{"URL":"https:\/\/elifesciences.org\/articles\/67995"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,7]]},"references-count":196,"alternative-id":["10.7554\/eLife.67995"],"URL":"http:\/\/dx.doi.org\/10.7554\/elife.67995","relation":{"references":[{"id-type":"doi","id":"10.31219\/osf.io\/5y8w7","asserted-by":"subject"}],"is-supplemented-by":[{"id-type":"accession","id":"e5nvr","asserted-by":"subject"}]},"ISSN":["2050-084X"],"subject":["General Immunology and Microbiology","General Biochemistry, Genetics and Molecular Biology","General Medicine","General Neuroscience"],"published":{"date-parts":[[2021,12,7]]}}' recorded_at: Wed, 05 Jul 2023 05:27:43 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.31234%2Fosf.io%2F5vfq6/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:44 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; rel="license" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:22Z","timestamp":1687134442960},"posted":{"date-parts":[[2019,12,18]]},"group-title":"PsyArXiv","reference-count":0,"publisher":"Center for Open Science","license":[{"start":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T00:00:00Z","timestamp":1576627200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"abstract":"

In light of the \u201creplication crisis,\u201d some advocate for stricter standards and greater transparency in research methods. These efforts push toward a data analysis approach called \u201cconfirmatory data analysis\u201d (CDA). However, some (e.g., Baumeister, 2016; Finkel, Eastwick, &amp; Reis, 2017; Goldin-Meadow, 2016) argued emphasizing CDA may restrict creativity and discovery. These scholars argued (sometimes inadvertently) for greater freedom to pursue an approach called \u201cexploratory data analysis\u201d (EDA). Unfortunately, many of those who push against stricter CDA standards don\u2019t realize EDA even exists, or fail to understand the philosophy and proper tools for exploration. In this paper, the meaning, tools, and ethics associated with EDA, CDA, and a relatively unknown but important approach called \u201crough CDA\u201d are clarified. Important distinctions are developed between EDA\/rough CDA\/CDA and problematic analysis activities including p-hacking, HARKing, and data mining. CDA, EDA, p-hacking, HARKing, etc., are situated in a unified (graphical) framework that clarifies ethical boundaries associated with each. In short, the proper data analytic approach depends on (1) intentions, and (2) transparency. Most psychological research isn\u2019t at a maturity level to justify CDA, and researchers have historically utilized tools that are mismatched to their research agenda. In the conclusion, guidelines are presented about how these typologies can be integrated into a cumulative research program that can help psychology move beyond the replication crisis.<\/p>","DOI":"10.31234\/osf.io\/5vfq6","type":"posted-content","created":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T19:35:52Z","timestamp":1576697752000},"source":"Crossref","is-referenced-by-count":3,"title":"Moving Beyond the \"Replication Crisis\": Understanding the Exploratory\/Confirmatory Data Analysis Continuum","prefix":"10.31234","author":[{"given":"Dustin","family":"Fife","sequence":"first","affiliation":[]},{"given":"Joseph Lee","family":"Rodgers","sequence":"additional","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T17:11:24Z","timestamp":1615223484000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/5vfq6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,18]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31234\/osf.io\/5vfq6","relation":{},"published":{"date-parts":[[2019,12,18]]},"subtype":"preprint"}' recorded_at: Wed, 05 Jul 2023 05:27:44 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1038%2Fs41562-021-01203-8/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:44 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="application/pdf"; rel="item", ; version="vor"; type="text/html"; rel="item", ; version="vor"; type="application/pdf"; rel="item", ; version="tdm"; rel="license", ; version="vor"; rel="license", ; title="Rink Hoekstra"; rel="author", ; title="Simine Vazire"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:22Z","timestamp":1687134442476},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T00:00:00Z","timestamp":1635379200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T00:00:00Z","timestamp":1635379200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000925","name":"John Templeton Foundation","doi-asserted-by":"publisher","award":["60937"]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"DOI":"10.1038\/s41562-021-01203-8","type":"journal-article","created":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T16:02:41Z","timestamp":1635436961000},"page":"1602-1607","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":"Aspiring to greater intellectual humility in science","prefix":"10.1038","volume":"5","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-1588-7527","authenticated-orcid":false,"given":"Rink","family":"Hoekstra","sequence":"first","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-3933-9752","authenticated-orcid":false,"given":"Simine","family":"Vazire","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,10,28]]},"reference":[{"key":"1203_CR1","doi-asserted-by":"publisher","unstructured":"[Editorial]. Tell it like it is. Nat. Hum. Behav. https:\/\/doi.org\/10.1038\/s41562-020-0818-9 (2020).","DOI":"10.1038\/s41562-020-0818-9"},{"key":"1203_CR2","unstructured":"Pashler, H. & De Ruiter, J. P. Taking responsibility for our field\u2019s reputation. APS Letter\/Observer Forum https:\/\/www.psychologicalscience.org\/observer\/taking-responsibility-for-our-fields-reputation (2017)."},{"key":"1203_CR3","doi-asserted-by":"publisher","unstructured":"Whitcomb, D., Battaly, H., Baehr, J. & Howard-Snyder, D. Intellectual humility: owning our limitations. Phil. Phenom. Res. https:\/\/doi.org\/10.1111\/phpr.12228 (2015).","DOI":"10.1111\/phpr.12228"},{"key":"1203_CR4","doi-asserted-by":"publisher","first-page":"e0182950","DOI":"10.1371\/journal.pone.0182950","volume":"12","author":"M Alfano","year":"2017","unstructured":"Alfano, M. et al. Development and validation of a multi-dimensional measure of intellectual humility. PLoS ONE 12, e0182950 (2017).","journal-title":"PLoS ONE"},{"key":"1203_CR5","doi-asserted-by":"publisher","first-page":"793","DOI":"10.1177\/0146167217697695","volume":"43","author":"MR Leary","year":"2017","unstructured":"Leary, M. R. et al. Cognitive and interpersonal features of intellectual humility. Pers. Soc. Psych. Bull. 43, 793\u2013813 (2017).","journal-title":"Pers. Soc. Psych. Bull."},{"key":"1203_CR6","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1080\/00223891.2015.1068174","volume":"98","author":"EJ Krumrei-Mancuso","year":"2015","unstructured":"Krumrei-Mancuso, E. J. & Rouse, S. V. The development and validation of the Comprehensive Intellectual Humility Scale. J. Pers. Assess. 98, 209\u2013221 (2015).","journal-title":"J. Pers. Assess."},{"key":"1203_CR7","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1177\/0963721419850153","volume":"28","author":"DR Van Tongeren","year":"2019","unstructured":"Van Tongeren, D. R., Davis, D. E., Hook, J. N. & Witvliet, C. V. Humility. Curr. Dir. Psych. Sci. 28, 463\u2013468 (2019).","journal-title":"Curr. Dir. Psych. Sci."},{"key":"1203_CR8","first-page":"115","volume":"1","author":"RK Merton","year":"1942","unstructured":"Merton, R. K. Science and technology in a democratic order. J. Leg. Pol. Soc. 1, 115\u2013126 (1942).","journal-title":"J. Leg. Pol. Soc."},{"key":"1203_CR9","doi-asserted-by":"publisher","first-page":"h6467","DOI":"10.1136\/bmj.h6467","volume":"351","author":"CH Vinkers","year":"2015","unstructured":"Vinkers, C. H., Tijdink, J. K. & Otte, W. M. Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. BMJ 351, h6467 (2015).","journal-title":"BMJ"},{"key":"1203_CR10","doi-asserted-by":"publisher","unstructured":"Yarkoni, T. The generalizability crisis. Behav. Brain Sci. https:\/\/doi.org\/10.1017\/S0140525X20001685 (2020).","DOI":"10.1017\/S0140525X20001685"},{"key":"1203_CR11","doi-asserted-by":"publisher","unstructured":"Riddle, T. Linguistic overfitting in empirical psychology. Preprint at https:\/\/doi.org\/10.31234\/osf.io\/qasde (2018).","DOI":"10.31234\/osf.io\/qasde"},{"key":"1203_CR12","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1525\/jer.2007.2.4.3","volume":"2","author":"MS Anderson","year":"2007","unstructured":"Anderson, M. S., Martinson, B. C. & De Vries, R. Normative dissonance in science: results from a national survey of US scientists. J. Empir. Res. Hum. Res. Ethics 2, 3\u201314 (2007).","journal-title":"J. Empir. Res. Hum. Res. Ethics"},{"key":"1203_CR13","doi-asserted-by":"publisher","first-page":"579","DOI":"10.2307\/2094423","volume":"39","author":"II Mitroff","year":"1974","unstructured":"Mitroff, I. I. Norms and counter-norms in a select group of the Apollo moon scientists: a case study of the ambivalence of scientists. Am. Soc. Rev. 39, 579\u2013595 (1974).","journal-title":"Am. Soc. Rev."},{"key":"1203_CR14","unstructured":"Sessions, R. How a \u2018difficult\u2019 composer gets that way. New York Times (8 January 1950)."},{"key":"1203_CR15","unstructured":"Bem, D. J. in The Compleat Academic: A Practical Guide for the Beginning Social Scientist (eds Darley, J. M. et al.) Ch. 10 (American Psychological Association, 2004)."},{"key":"1203_CR16","unstructured":"Kail, R. V. Reflections on five years as editor. APS Observer https:\/\/www.psychologicalscience.org\/observer\/reflections-on-five-years-as-editor (2012)."},{"key":"1203_CR17","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1126\/science.aab2374","volume":"348","author":"BA Nosek","year":"2015","unstructured":"Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422\u20131425 (2015).","journal-title":"Science"},{"key":"1203_CR18","doi-asserted-by":"publisher","first-page":"556","DOI":"10.1177\/2515245918806489","volume":"1","author":"WE Davis","year":"2018","unstructured":"Davis, W. E. et al. Peer-review guidelines promoting replicability and transparency in psychological science. Adv. Methods Pract. Psychol. Sci. 1, 556\u2013573 (2018).","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"1203_CR19","doi-asserted-by":"publisher","first-page":"150547","DOI":"10.1098\/rsos.150547","volume":"3","author":"RD Morey","year":"2016","unstructured":"Morey, R. D. et al. The Peer Reviewers\u2019 Openness Initiative: incentivizing open research practices through peer review. R. Soc. Open Sci. 3, 150547 (2016).","journal-title":"R. Soc. Open Sci."},{"key":"1203_CR20","first-page":"4","volume":"26","author":"JP Simmons","year":"2012","unstructured":"Simmons, J. P., Nelson, L. & Simonsohn, U. A 21 word solution. Soc. Pers. Soc. Psych. Dial. 26, 4\u20137 (2012).","journal-title":"Soc. Pers. Soc. Psych. Dial."},{"key":"1203_CR21","doi-asserted-by":"publisher","first-page":"609","DOI":"10.1016\/j.cortex.2012.12.016","volume":"49","author":"CD Chambers","year":"2013","unstructured":"Chambers, C. D. Registered Reports: a new publishing initiative at Cortex. Cortex 49, 609\u2013610 (2013).","journal-title":"Cortex"},{"key":"1203_CR22","doi-asserted-by":"publisher","first-page":"250","DOI":"10.3758\/BF03204506","volume":"25","author":"GR Loftus","year":"1993","unstructured":"Loftus, G. R. A picture is worth a thousand P values: on the irrelevance of hypothesis testing in the microcomputer age. Behav. Res. Meth. Inst. Comp. 25, 250\u2013256 (1993).","journal-title":"Behav. Res. Meth. Inst. Comp."},{"key":"1203_CR23","doi-asserted-by":"crossref","unstructured":"Dragicevic, P., Jansen, Y., Sarma, A., Kay, M. & Chevalier, F. Increasing the transparency of research papers with explorable multiverse analyses. In Proc. 2019 Conf. on Human Factors in Computing Systems (eds Brewster, S. et al.) Paper no. 65 (Association for Computing Machinery, 2019).","DOI":"10.1145\/3290605.3300295"},{"key":"1203_CR24","doi-asserted-by":"publisher","first-page":"702","DOI":"10.1177\/1745691616658637","volume":"11","author":"S Steegen","year":"2016","unstructured":"Steegen, S., Tuerlinckx, F., Gelman, A. & Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11, 702\u2013712 (2016).","journal-title":"Perspect. Psychol. Sci."},{"key":"1203_CR25","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1177\/1745691612463078","volume":"7","author":"EJ Wagenmakers","year":"2012","unstructured":"Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. & Kievit, R. A. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632\u2013638 (2012).","journal-title":"Perspect. Psychol. Sci."},{"key":"1203_CR26","doi-asserted-by":"publisher","first-page":"1033","DOI":"10.3758\/BF03213921","volume":"13","author":"R Hoekstra","year":"2006","unstructured":"Hoekstra, R., Finch, S., Kiers, H. A. L. & Johnson, A. Probability as certainty: dichotomous thinking and the misuse of P values. Psychon. Bull. Rev. 13, 1033\u20131037 (2006).","journal-title":"Psychon. Bull. Rev."},{"key":"1203_CR27","doi-asserted-by":"publisher","first-page":"181870","DOI":"10.1098\/rsos.181870","volume":"6","author":"AM Van der Bles","year":"2019","unstructured":"Van der Bles, A. M. et al. Communicating uncertainty about facts, numbers and science. R. Soc. Open Sci. 6, 181870 (2019).","journal-title":"R. Soc. Open Sci."},{"key":"1203_CR28","doi-asserted-by":"crossref","first-page":"103","DOI":"10.54870\/1551-3440.1338","volume":"12","author":"R Hoekstra","year":"2015","unstructured":"Hoekstra, R. Risk as an explanatory factor for researchers\u2019 inferential interpretations. Math. Enthus. 12, 103\u2013112 (2015).","journal-title":"Math. Enthus."},{"key":"1203_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/00031305.2019.1583913","volume":"73","author":"RL Wasserstein","year":"2019","unstructured":"Wasserstein, R. L., Schirm, A. L. & Lazar, N. A. Moving to a world beyond \u2018P < 0.05\u2019. Am. Stat. 73, 1\u201319 (2019).","journal-title":"Am. Stat."},{"key":"1203_CR30","doi-asserted-by":"publisher","first-page":"5","DOI":"10.3758\/s13423-017-1262-3","volume":"25","author":"A Etz","year":"2018","unstructured":"Etz, A. & Vandekerckhove, J. Introduction to Bayesian inference for psychology. Psychon. Bull. Rev. 25, 5\u201334 (2018).","journal-title":"Psychon. Bull. Rev."},{"key":"1203_CR31","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1177\/2515245918779348","volume":"1","author":"A Ly","year":"2018","unstructured":"Ly, A. et al. Bayesian reanalyses from summary statistics: a guide for academic consumers. Adv. Methods Pract. Psychol. Sci. 1, 367\u2013374 (2018).","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"1203_CR32","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/1948550615603955","volume":"7","author":"S Vazire","year":"2016","unstructured":"Vazire, S. Editorial. Soc. Psychol. Personal. Sci. 7, 3\u20137 (2016).","journal-title":"Soc. Psychol. Personal. Sci."},{"key":"1203_CR33","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.1177\/1745691617708630","volume":"12","author":"DJ Simons","year":"2017","unstructured":"Simons, D. J., Shoda, Y. & Lindsay, D. S. Constraints on generality (COG): a proposed addition to all empirical papers. Perspect. Psychol. Sci. 12, 1123\u20131128 (2017).","journal-title":"Perspect. Psychol. Sci."},{"key":"1203_CR34","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1038\/s41562-020-00990-w","volume":"4","author":"H IJzerman","year":"2020","unstructured":"IJzerman, H. et al. Use caution when applying behavioural science to policy. Nat. Hum. Behav. 4, 1092\u20131094 (2020).","journal-title":"Nat. Hum. Behav."},{"key":"1203_CR35","doi-asserted-by":"publisher","unstructured":"Flake, J. K., & Fried, E. I. Measurement schmeasurement: questionable measurement practices and how to avoid them. Preprint at https:\/\/doi.org\/10.31234\/osf.io\/hs7wm (2019).","DOI":"10.31234\/osf.io\/hs7wm"},{"key":"1203_CR36","unstructured":"Schubert, S. Hedge-drift and advanced motte-and-bailey. LessWrong Blog https:\/\/www.lesswrong.com\/posts\/oMYeJrQmCeoY5sEzg\/hedge-drift-and-advanced-motte-and-bailey (2016)."},{"key":"1203_CR37","doi-asserted-by":"publisher","first-page":"g7015","DOI":"10.1136\/bmj.g7015","volume":"349","author":"P Sumner","year":"2014","unstructured":"Sumner, P. et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ 349, g7015 (2014).","journal-title":"BMJ"},{"key":"1203_CR38","doi-asserted-by":"publisher","unstructured":"Rohrer, J. M. et al. Putting the self in self-correction: findings from the loss-of-confidence project. Perspect. Psychol. Sci., https:\/\/doi.org\/10.1177\/1745691620964106 (2021).","DOI":"10.1177\/1745691620964106"},{"key":"1203_CR39","unstructured":"Yarkoni, T. (2018). No, it\u2019s not the incentives\u2014it\u2019s you. Talyarkoni Blog https:\/\/www.talyarkoni.org\/blog\/2018\/10\/02\/no-its-not-the-incentives-its-you\/ (2018)."}],"container-title":"Nature Human Behaviour","original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41562-021-01203-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41562-021-01203-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41562-021-01203-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T00:32:07Z","timestamp":1670027527000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41562-021-01203-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,28]]},"references-count":39,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["1203"],"URL":"http:\/\/dx.doi.org\/10.1038\/s41562-021-01203-8","relation":{},"ISSN":["2397-3374"],"subject":["Behavioral Neuroscience","Experimental and Cognitive Psychology","Social Psychology"],"container-title-short":"Nat Hum Behav","published":{"date-parts":[[2021,10,28]]},"assertion":[{"value":"12 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 August 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}' recorded_at: Wed, 05 Jul 2023 05:27:44 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1177%2F0956797611430953/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:45 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="application/pdf"; rel="item", ; version="vor"; type="application/xml"; rel="item", ; version="vor"; rel="item", ; version="tdm"; rel="license" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,7,4]],"date-time":"2023-07-04T20:09:40Z","timestamp":1688501380479},"reference-count":31,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2012,4,16]],"date-time":"2012-04-16T00:00:00Z","timestamp":1334534400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"published-print":{"date-parts":[[2012,5]]},"abstract":" Cases of clear scientific misconduct have received significant media attention recently, but less flagrantly questionable research practices may be more prevalent and, ultimately, more damaging to the academic enterprise. Using an anonymous elicitation format supplemented by incentives for honest reporting, we surveyed over 2,000 psychologists about their involvement in questionable research practices. The impact of truth-telling incentives on self-admissions of questionable research practices was positive, and this impact was greater for practices that respondents judged to be less defensible. Combining three different estimation methods, we found that the percentage of respondents who have engaged in questionable practices was surprisingly high. This finding suggests that some questionable practices may constitute the prevailing research norm. <\/jats:p>","DOI":"10.1177\/0956797611430953","type":"journal-article","created":{"date-parts":[[2012,4,17]],"date-time":"2012-04-17T05:55:06Z","timestamp":1334642106000},"page":"524-532","update-policy":"http:\/\/dx.doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1181,"title":"Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling","prefix":"10.1177","volume":"23","author":[{"given":"Leslie K.","family":"John","sequence":"first","affiliation":[{"name":"Marketing Unit, Harvard Business School"}]},{"given":"George","family":"Loewenstein","sequence":"additional","affiliation":[{"name":"Department of Social & Decision Sciences, Carnegie Mellon University"}]},{"given":"Drazen","family":"Prelec","sequence":"additional","affiliation":[{"name":"Sloan School of Management and Departments of Economics and Brain & Cognitive Sciences, Massachusetts Institute of Technology"}]}],"member":"179","published-online":{"date-parts":[[2012,4,16]]},"reference":[{"key":"bibr2-0956797611430953","doi-asserted-by":"publisher","DOI":"10.3758\/BF03329773"},{"key":"bibr3-0956797611430953","volume-title":"Improving interview method and questionnaire design: Response effects to threatening questions in survey research","author":"Bradburn N.","year":"1979"},{"key":"bibr4-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1126\/science.284.5420.1670"},{"key":"bibr5-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1038\/479151a"},{"key":"bibr6-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c5347"},{"key":"bibr7-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1509\/jmkr.47.1.14"},{"key":"bibr9-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1126\/science.284.5420.1599a"},{"key":"bibr12-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2004.11.013"},{"key":"bibr13-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1136\/jme.27.5.344"},{"key":"bibr14-0956797611430953","first-page":"142","volume-title":"Scaling: A sourcebook for behavioral scientists","author":"Guttman L. L.","year":"1974"},{"key":"bibr15-0956797611430953","doi-asserted-by":"publisher","DOI":"10.5694\/j.1326-5377.2005.tb06813.x"},{"key":"bibr16-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1001\/jama.294.2.218"},{"key":"bibr18-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.108.3.480"},{"key":"bibr19-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1093\/jleo\/ewp031"},{"key":"bibr21-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1177\/0049124104268664"},{"key":"bibr22-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1093\/ei\/39.1.162"},{"key":"bibr23-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1126\/science.290.5497.1662"},{"key":"bibr24-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1038\/435737a"},{"key":"bibr25-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ecolsys.31.1.441"},{"key":"bibr26-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1126\/science.1102081"},{"key":"bibr28-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611417632"},{"key":"bibr29-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1007\/s11673-008-9113-6"},{"key":"bibr30-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9280.00169"},{"key":"bibr31-0956797611430953","first-page":"1","volume-title":"Investigating Research Integrity: Proceedings of the First ORI Research Conference on Research Integrity","author":"Steneck N. H.","year":"2002"},{"key":"bibr32-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1007\/s11948-006-0006-y"},{"key":"bibr33-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1207\/s15327019eb1604_3"},{"key":"bibr34-0956797611430953","first-page":"542","volume":"81","author":"Swazey J. P.","year":"1993","journal-title":"American Scientist"},{"key":"bibr35-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.133.5.859"},{"key":"bibr36-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1126\/science.280.5365.867"},{"key":"bibr37-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1965.10480775"},{"key":"bibr38-0956797611430953","doi-asserted-by":"publisher","DOI":"10.1038\/480007a"}],"container-title":"Psychological Science","original-title":[],"language":"en","link":[{"URL":"http:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0956797611430953","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0956797611430953","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0956797611430953","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,5]],"date-time":"2020-11-05T21:17:52Z","timestamp":1604611072000},"score":1,"resource":{"primary":{"URL":"http:\/\/journals.sagepub.com\/doi\/10.1177\/0956797611430953"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,4,16]]},"references-count":31,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2012,5]]}},"alternative-id":["10.1177\/0956797611430953"],"URL":"http:\/\/dx.doi.org\/10.1177\/0956797611430953","relation":{},"ISSN":["0956-7976","1467-9280"],"subject":["General Psychology"],"container-title-short":"Psychol Sci","published":{"date-parts":[[2012,4,16]]}}' recorded_at: Wed, 05 Jul 2023 05:27:45 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1207%2Fs15327965pli0102_1/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:45 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; rel="item" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:24Z","timestamp":1687134444877},"reference-count":114,"publisher":"Informa UK Limited","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"published-print":{"date-parts":[[1990,4]]},"DOI":"10.1207\/s15327965pli0102_1","type":"journal-article","created":{"date-parts":[[2006,3,17]],"date-time":"2006-03-17T08:48:55Z","timestamp":1142585335000},"page":"108-141","source":"Crossref","is-referenced-by-count":328,"title":"Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It","prefix":"10.1080","volume":"1","author":[{"given":"Paul E.","family":"Meehl","sequence":"first","affiliation":[]}],"member":"301","reference":[{"key":"p_1_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1037\/h0093360","volume":"47","author":"Allport G. W.","year":"1936","journal-title":"Psychological Monographs"},{"key":"p_2_2","unstructured":"Andreski, S. (1972). Social sciences as sorcery. London: Deutsch."},{"key":"p_3_3","doi-asserted-by":"crossref","first-page":"227","DOI":"10.3758\/BF03342263","volume":"4","author":"Aronson E.","year":"1966","journal-title":"Psychonomic Science"},{"key":"p_4_4","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1037\/h0020412","volume":"66","author":"Bakan D.","year":"1966","journal-title":"Psychological Bulletin"},{"key":"p_5_5","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1126\/science.134.3479.596","volume":"134","author":"Barber B.","year":"1961","journal-title":"Science"},{"key":"p_6_6","unstructured":"Betz, N. E. (Ed.). (1986). The g factor in employment [Special issue]. Journal of Vocational Behavior, 29(3)."},{"key":"p_7_7","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1093\/bjps\/38.4.533","volume":"38","author":"Brink C.","year":"1987","journal-title":"British Journal for Philosophy of Science"},{"key":"p_8_8","doi-asserted-by":"publisher","DOI":"10.1086\/286432"},{"key":"p_9_9","doi-asserted-by":"crossref","first-page":"513","DOI":"10.2307\/2102817","volume":"5","author":"Carnap R.","year":"1945","journal-title":"Philosophy and Phenomenological Research"},{"key":"p_10_10","unstructured":"Carnap, R. (1966). Philosophical foundations of physics. New York: Bas~c."},{"key":"p_11_11","doi-asserted-by":"crossref","unstructured":"Cartwright, N. (1983). How the laws of physics lie. New York: Oxford University Press.","DOI":"10.1093\/0198247044.001.0001"},{"key":"p_12_12","doi-asserted-by":"crossref","first-page":"378","DOI":"10.17763\/haer.48.3.t490261645281841","volume":"48","author":"Carver R. P.","year":"1978","journal-title":"Harvard Educational Review"},{"key":"p_13_13","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1080\/00223980.1987.9712666","volume":"121","author":"Chow S. L.","year":"1987","journal-title":"Journal of Psychology"},{"key":"p_14_14","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.103.1.105"},{"key":"p_15_15","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1037\/h0040957","volume":"52","author":"Cronbach L. J.","year":"1955","journal-title":"Psychological Bulletin"},{"key":"p_16_16","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.42.2.145"},{"key":"p_17_17","unstructured":"Dawes, R. M. (1988). Rational choice in an uncertain world. Chicago: Harcourt Brace Jovanovich."},{"key":"p_18_18","doi-asserted-by":"crossref","first-page":"1668","DOI":"10.1126\/science.2648573","volume":"243","author":"Dawes R. M.","year":"1989","journal-title":"Science"},{"key":"p_19_19","unstructured":"Draper, N. & Smith, H. (1981). Applied regression analysis (2nd ed.). New York: Wiley."},{"key":"p_20_20","doi-asserted-by":"crossref","first-page":"14","DOI":"10.2307\/1598947","volume":"35","author":"Dworkin R. M.","year":"1967","journal-title":"Universtty of Chicago Law Review"},{"key":"p_21_21","unstructured":"Eisberg, R. M. (1961). Fundamentals of modern physics. New York: W~ley."},{"key":"p_22_22","first-page":"1","volume":"12","author":"Evans C.","year":"1941","journal-title":"Journal of Psychology"},{"key":"p_23_23","unstructured":"Faust, D. (1984). The limits of scientific reasoning. Minneapolis: University of Minnesota Press."},{"key":"p_24_24","unstructured":"Feyerabend, P. (1970). Against method. In M. Radner & S. Winokur (Eds.), Minnesota studies in thephilosophy of science: Vol. IV. Analyses of theories and methods of physics andpsychology (pp. 17-130). Minneapolis: University of Minnesota Press."},{"key":"p_25_25","unstructured":"Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver & Boyd."},{"key":"p_26_26","unstructured":"Fisher, R. A. (1937). The design of experiments (2nd ed.). London: Oliver & Boyd."},{"key":"p_27_27","unstructured":"Fiske, D. W. & Shweder, R. A. (1986). Metatheory in social science. Chicago: University of Chicago Press."},{"key":"p_28_28","unstructured":"Freud, S. (1957). On the history of the psycho-analytic movement. In J. Strachey (Ed. &Trans.), The standard edition of the completepsychological works of SigmundFreud (Vol. 14, pp. 7-66). London: Hogarth. (Original work published 1914)"},{"key":"p_29_29","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1038\/scientificamerican0383-124","volume":"248","author":"Geller S. A.","year":"1983","journal-title":"Autopsy. Scientific American"},{"key":"p_30_30","unstructured":"Giere, R. N. (1984). Understanding scientific reasoning. New York: Holt, Rinehart & Winston."},{"key":"p_31_31","doi-asserted-by":"crossref","unstructured":"Giere, R. N. (1988). Explaining science: A cognitive approach. Chicago: University of Chicago Press.","DOI":"10.7208\/chicago\/9780226292038.001.0001"},{"key":"p_32_32","unstructured":"Glass, G. V., Cahen, L. S., Smith, M. L. & Filby, N. K. (1982). School class size: Research andpolicy. Beverly Hills, CA: Sage."},{"key":"p_33_33","unstructured":"Glass, G. V., McGaw, B. &Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage."},{"key":"p_34_34","unstructured":"Glymour, C. (1980). Theory and evidence. Princeton, NJ: Princeton University Press."},{"key":"p_35_35","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1111\/j.1469-1809.1978.tb00924.x","volume":"41","author":"Golden R.","year":"1978","journal-title":"Annals of Human Genetzcs London"},{"key":"p_36_36","doi-asserted-by":"publisher","DOI":"10.1086\/289462"},{"key":"p_37_37","unstructured":"Gough, H. G. (1987). CPI, Administrator''s guide. Palo Alto, CA: Consulting Psychologists Press."},{"key":"p_38_38","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1093\/bjps\/39.3.277","volume":"39","author":"Hacking I.","year":"1988","journal-title":"British Journal for the Philosophy of Science"},{"key":"p_39_39","unstructured":"Harman, H. H. (1960). Modern factor analysis. Chicago: University of Chicago Press."},{"key":"p_40_40","doi-asserted-by":"publisher","DOI":"10.1037\/\/0003-066X.42.5.443"},{"key":"p_41_41","unstructured":"Hodgman, C. D. (Compiler). (1941). Mathematical tables from handbook of chemistry and physics (7th ed.). Cleveland, OH: Chemical Rubber Publishing Co."},{"key":"p_42_42","unstructured":"Hogarth, R. M. (1987). Judgment andchoice: Thepsychology ofdecision. New York: Wiley."},{"key":"p_43_43","unstructured":"Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century-Crofts."},{"key":"p_44_44","unstructured":"Hunter, J. E., Schmidt, F. L. & Jackson, G. B. (1982). Meta-analysis: Cumulating researchfindings across studies. Beverly Hills, CA: Sage."},{"key":"p_45_45","doi-asserted-by":"crossref","unstructured":"Kahneman, D., Slovic, P. & Tversky, A. (Eds.). (1982). Judgments under uncertainty: Heuristics and biases. Cambridge, England: Cambridge University Press.","DOI":"10.1017\/CBO9780511809477"},{"key":"p_46_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/030631283013001003","volume":"13","author":"Kern L. H.","year":"1983","journal-title":"Social Studies of Science"},{"key":"p_47_47","unstructured":"Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.; Vol. 2, No. 2 of International Encyclopedia of Unzjied Science). Chicago: University of Chicago Press."},{"key":"p_48_48","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1093\/aristotelian\/69.1.149","volume":"69","author":"Lakatos","year":"1968","journal-title":"Proceedings of the Aristotelian Society"},{"key":"p_49_49","unstructured":"Lakatos, I. (1 970). Falsification and the methodology of scientific research programmes. In I. Lakatos & A. Musgrave (Eds.), Criticism and the growth of knowledge (pp. 91-195). Cambridge, England: Cambridge University Press. (Reprinted in J. Worrall & G. Currie [Eds.], Imre Lakatos: Philosophical papers. Vol. I: The methodology of scientific research programmes [pp. 8- 1011. New York: Cambridge University Press, 1978)"},{"key":"p_50_50","doi-asserted-by":"crossref","unstructured":"Lakatos, I. (1971). History of science and its rational reconstructions. In R. C. Buck & R. S. Cohen (Eds.), P. S. A. (1970 Boston Studies in the Philosophy of Science, Vol. 8, pp. 91-135). Dordrecht, Netherlands: Reidel. (Reprinted in J. Worrall & G. Currie [Eds.], Imre Lakatos: Philosophical papers. Vol. I: The methodology of scientific research programmes [pp. 102-1381. New York: Cambridge University Press, 1978)","DOI":"10.1007\/978-94-010-3142-4_7"},{"key":"p_51_51","doi-asserted-by":"crossref","first-page":"1249","DOI":"10.1056\/NEJM198805123181906","volume":"318","author":"Landefeld C. S.","year":"1983","journal-title":"New England Journal of Medicine"},{"key":"p_53_52","unstructured":"Laudan, L. (1977). Progress and its problems: Toward a theory of scientific growth. Berkeley: University of California Press."},{"key":"p_54_53","doi-asserted-by":"crossref","first-page":"427","DOI":"10.2307\/2023861","volume":"67","author":"Lewis D.","year":"1970","journal-title":"Journal of Philosophy"},{"key":"p_55_54","doi-asserted-by":"publisher","DOI":"10.1037\/\/0022-3514.37.6.833"},{"key":"p_56_55","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1037\/h0026141","volume":"70","author":"Lykken D. T.","year":"1968","journal-title":"Psychological Bulletin"},{"key":"p_57_56","unstructured":"MacCorquodale, K. & Meehl, P. E. (1954). E. C. Tolman. In W. K. Estes, S. Koch, K. MacCorquodale, P. E. Meehl, C. G. Mueller, W. N. Schoenfeld, & W. S. Verplanck, Modern learning theory (pp. 177-266). New York: Appleton-Century-Crofts."},{"key":"p_58_57","unstructured":"Mahoney, M. J. (1976). Scientist a s subject: The psychological imperative. Cambridge, MA: Ballinger."},{"key":"p_59_58","unstructured":"Maxwell, G. (1962). The ontological status of theoretical entities. In H. Feigl & G. Maxwell (Eds.), Minnesota studies in the philosophy of science: Vol. 3. Scientific explanations, space, and time (pp. 3-27). Minneapolis: University of Minnesota Press."},{"key":"p_60_59","unstructured":"Maxwell, G. (1970). Structural realism and the meaning of theoretical terms. In M. Radner & S. Winokur (Eds.), Minnesota studies in the philosophy of science: Vol. 4. Analyses of theories and methods of physics and psychology (pp. 18 1- 192). Minneapolis: University of Minnesota Press."},{"key":"p_62_60","doi-asserted-by":"crossref","unstructured":"Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.","DOI":"10.1037\/11281-000"},{"key":"p_63_61","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1037\/h0041029","volume":"17","author":"Meehl P. E.","year":"1962","journal-title":"American Psychologist"},{"key":"p_64_62","unstructured":"Meehl, P. E. (1964). Minnesota-FordProjectgenotypicpersonality items [IBM and compatibles machine-readable data file]. (Available, with the Minnesota-Ford pool of phenotypic personality items, from P. E. Meehl, Department of Psychology, N218 Elliott Hall, University of Minnesota, 75 East River Road, MimeapoIis, MN 55455)"},{"key":"p_65_63","doi-asserted-by":"publisher","DOI":"10.1086\/288135"},{"key":"p_66_64","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1037\/h0030750","volume":"77","author":"Meehl P. E.","year":"1971","journal-title":"Journal of Abnormal Psychology"},{"key":"p_67_65","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1080\/00207411.1972.11448562","volume":"1","author":"Meehl P. E.","year":"1972","journal-title":"International Journal of Mental Health"},{"key":"p_68_66","unstructured":"Meehl, P. E. (1973a). MAXCOV-HITMAX: A taxonomic search method for loose genetic syndromes. In P. E. Meehl, Psychodiagnosis: Selected papers (pp. 200-224). Minneapolis: University of Minnesota Press."},{"key":"p_69_67","unstructured":"Meehl, P. E. (1973b). Psychodiagnosis: Selected papers. Minneapolis: University of Minnesota Press."},{"key":"p_70_68","doi-asserted-by":"publisher","DOI":"10.1037\/\/0022-006X.46.4.806"},{"key":"p_71_69","doi-asserted-by":"crossref","unstructured":"Meehl, P. E. (1983a). Consistency tests in estimating the completeness of the fossil record: A neo-Popperian approach to statistical paleontology. In J. Earman (Ed.), Minnesota studies in thephilosophy of science: Vol. X. Testing scientific theories (pp. 413-473). Minneapolis: University of Minnesota Press.","DOI":"10.5749\/j.cttts94f.18"},{"key":"p_72_70","doi-asserted-by":"crossref","unstructured":"Meehl, P. E. (1983b). Subjectivity in psychoanalytic inference: The nagging persistence of Wilhelm Fliess''s Achensee question. In J. Earman ( ~ d ), Minnesota studies in the philosophy of science: Vol. X. Testing scientific theories (pp. 349-41 1). Minneapolis: University of Minnesota Press.","DOI":"10.5749\/j.cttts94f.17"},{"key":"p_73_71","unstructured":"Meehl, P. E. (1984). Foreword. In D. Faust, The limitsof scientific reasoning (pp. xi-xxiv). Minneapolis: University of Minnesota Press."},{"key":"p_74_72","doi-asserted-by":"publisher","DOI":"10.1207\/s15327752jpa5003_6"},{"key":"p_75_73","unstructured":"Meehl, P. E. (1986b). Diagnostic taxa as open concepts: Metatheoretical and statistical questions about reliability and construct validity in the grand strategy of nosological revision. In T. Millon & G. L. Klerman (Eds.), Contemporary directions in psychopathology (pp. 215-23 1). New York: Guilford."},{"key":"p_76_74","doi-asserted-by":"crossref","first-page":"935","DOI":"10.1001\/archpsyc.1989.01810100077015","volume":"46","author":"Meehl P. E.","year":"1989","journal-title":"Archives of General Psychiatry"},{"key":"p_77_75","unstructured":"Meehl, P. E. (1990a). Corroboration and verisimilitude: Against Lakatos'' \"sheer leap offaith\" (Working paper). Minneapolis: Minnesota Center for Philosophy of Science."},{"key":"p_78_76","unstructured":"Meehl, P. E. (1990b). Schizotaxia as an open concept. In A. I. Rahin, R. Zucker, R. Emmons, & S. Frank (Eds.), Studying persons and lives (pp. 248-303). New York: Springer."},{"key":"p_79_77","doi-asserted-by":"publisher","DOI":"10.2466\/PR0.66.1.195-244"},{"key":"p_80_78","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1521\/pedi.1990.4.1.1","volume":"4","author":"Meehl P. E.","year":"1990","journal-title":"Journal of Personality Disorders"},{"key":"p_81_79","unstructured":"Meehl, P. E. &Golden, R. (1982). Taxometric methods. In P. Kendall & J. Butcher (Eds.), Handbook of research methods in clinicalpsychology (pp. 127-181). New York: Wiley."},{"key":"p_82_80","first-page":"171","volume":"5","author":"Meehl P. E.","year":"1971","journal-title":"Journal of Experimental Research in Personality"},{"key":"p_83_81","unstructured":"Meehl, P. E., Schofield, W., Glueck, B. C., Studdiford, W. B., Hastings, D. W., Hathaway, S. R. & Clyde, D. J. (1962). Minnesota-Ford pool of phenotypic personality items (August 1962 ed.). Minneapolis: University of Minnesota."},{"key":"p_84_82","unstructured":"Morrison, D. E. & Henkel, R. E. (Eds.). (1970). The significance test controversy. Chicago: Aldine."},{"key":"p_85_83","unstructured":"Neurath, 0. (1932-1933). Protokollsatze. Erkenntnis, 3. (Trans. as \"Protocol sentences\" by F. Schick in A. J. Ayer [Ed.], LogicaEpositivism [pp. 201-2041, Glencoe, IL: Free Press, 1959)"},{"key":"p_86_84","unstructured":"Nisbett, R. E. &Ross, L. (1980). Human inference: Strategies and shortcomings of human judgment. Englewood Cliffs, NJ: Prentice-Hall."},{"key":"p_87_85","unstructured":"Nye, M. J. (1972). Molecular reality. London: Macdonald."},{"key":"p_88_86","unstructured":"O''Hear, A. (1980). Karl Popper. Boston: Routledge & Kegan Paul."},{"key":"p_89_87","first-page":"3","volume":"5","author":"Pap A.","year":"1953","journal-title":"Methodos"},{"key":"p_90_88","unstructured":"Pap, A. (1958). Semantics and necessary truth. New Haven, CT: Yale University Press."},{"key":"p_91_89","unstructured":"Pap, A. (1962). An introduction to the philosophy of science. New York: Free Press."},{"key":"p_92_90","first-page":"510","volume":"32","author":"Peppard T. A.","year":"1949","journal-title":"Minnesota Medicine"},{"key":"p_93_91","unstructured":"Penin, J. B. (1916). Atoms (D. L. Hammick, Trans.). New York: Van Nostrand. (Original work published 1913)"},{"key":"p_94_92","doi-asserted-by":"crossref","unstructured":"Popper, K. R. (1959). The logic of scientific discovery. New York: Basic. (Original work published 1935)","DOI":"10.1063\/1.3060577"},{"key":"p_95_93","unstructured":"Popper, K. R. (1962). Conjectures and refutations. New York: Basic."},{"key":"p_96_94","unstructured":"Popper, K. R. (1983). Postscript to the logic of scientific discovery: Vol. I . Realism and the aim of science W. W. Bartley I11 (Ed.). Totowa, NJ: Rowman & Littlefield."},{"key":"p_97_95","unstructured":"Rawls, J. (1971). A theory ofjustice. Cambridge, MA: Harvard University Press, Belknap Press."},{"key":"p_98_96","unstructured":"Reichenbach, H. (1938). Experience andprediction. Chicago: University of Chicago Press."},{"key":"p_99_97","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1037\/h0042040","volume":"57","author":"Rozehoom W. W.","year":"1960","journal-title":"Psychological Bulletin"},{"key":"p_100_98","unstructured":"Salmon, W. C . (1984). Scientijic explanation and the causal structure of the world. Princeton, NJ: Princeton University Press."},{"key":"p_101_99","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1037\/h0023624","volume":"66","author":"Sawyer J.","year":"1966","journal-title":"Psychological Bulletin"},{"key":"p_102_100","unstructured":"Schilpp, P. A. (Ed.). (1974). Thephilosophy of Karl Popper. LaSalle, IL: Open Court."},{"key":"p_103_101","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1037\/0003-066X.40.1.73","volume":"40","author":"Serlin R. C.","year":"1985","journal-title":"American Psychologist"},{"key":"p_104_102","first-page":"80","volume":"40","author":"Simon H. A.","year":"1945","journal-title":"Journal of the American Statistical Association"},{"key":"p_105_103","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1192\/bjp.116.531.129","volume":"116","author":"Sines J.","year":"1970","journal-title":"British Journal of Psychiatry"},{"key":"p_106_104","unstructured":"Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. New York: Appleton-Century."},{"key":"p_107_105","doi-asserted-by":"publisher","DOI":"10.1037\/\/0003-066X.32.8.638"},{"key":"p_108_106","unstructured":"Smith, M . L., Glass, G . V. &Miller, T. I. (1980). The benefitsofpsychotherapy. Baltimore: Johns Hopkins University Press."},{"key":"p_109_107","unstructured":"Suppe, F. (Ed.). (1977). The structure of scientific theories (2nd ed.). Urbana: University of Illinois Press."},{"key":"p_110_108","unstructured":"Taton, R. (1957). Reason and chance in scienti$c discovery. London. Hutchinson."},{"key":"p_111_109","unstructured":"Thurstone, L. L. (1938). Primary mental abilities. Chicago: University of Chicago Press."},{"key":"p_112_110","unstructured":"Thurstone, L. L. & Thurstone, T. G . (1941). Factorial studies of intelligence. Chicago: University of Chicago Press."},{"key":"p_113_111","unstructured":"Worrall, J. & Currie, G. (Eds.). (1978a). Imre Lakatos: Philosophical papers. Vol. I : The methodology of scientific research programmes. New York: Cambridge University Press."},{"key":"p_114_112","unstructured":"Worrall, J. & Cunie, G. (Eds.). (1978b). Imre Lakatos: Philosophical papers.Vol. 2: Mathematics, science and epistemology. New York: Cambridge University Press."},{"key":"p_115_113","unstructured":"Wren, P. C. (1925). Beau Geste. New York: Frederick A. Stokes."},{"key":"p_116_114","unstructured":"Ziman, J. (1978). Reliable knowledge. New York: Cambridge University Press."}],"container-title":"Psychological Inquiry","original-title":[],"language":"en","link":[{"URL":"http:\/\/www.tandfonline.com\/doi\/pdf\/10.1207\/s15327965pli0102_1","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,4,17]],"date-time":"2019-04-17T07:35:32Z","timestamp":1555486532000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.tandfonline.com\/doi\/abs\/10.1207\/s15327965pli0102_1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[1990,4]]},"references-count":114,"journal-issue":{"issue":"2","published-print":{"date-parts":[[1990,4]]}},"alternative-id":["10.1207\/s15327965pli0102_1"],"URL":"http:\/\/dx.doi.org\/10.1207\/s15327965pli0102_1","relation":{},"ISSN":["1047-840X","1532-7965"],"subject":["General Psychology"],"container-title-short":"Psychological Inquiry","published":{"date-parts":[[1990,4]]}}' recorded_at: Wed, 05 Jul 2023 05:27:46 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.31234%2Fosf.io%2Fwxn58/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:46 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; rel="license", ; title="Danielle Navarro"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T16:14:55Z","timestamp":1687277695569},"posted":{"date-parts":[[2020,9,23]]},"group-title":"PsyArXiv","reference-count":0,"publisher":"Center for Open Science","license":[{"start":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T00:00:00Z","timestamp":1600819200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"abstract":"

This is an archived version of a blog post on preregistration. The first half of the post argues that there is not a strong justification for preregistration as a tool to solve problems with statistical inference (p-hacking); the second half argues that preregistration has a stronger justification as one tool (among many) that can aid scientists in documenting our projects. [Note that this archival version exists only because the blog itself no longer does, and as the original has been cited multiple times there is value in ensuring that some version of the blog post remains accessible.]<\/p>","DOI":"10.31234\/osf.io\/wxn58","type":"posted-content","created":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T06:34:16Z","timestamp":1600842856000},"source":"Crossref","is-referenced-by-count":7,"title":"Paths in strange spaces: A comment on preregistration","prefix":"10.31234","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-7648-6578","authenticated-orcid":true,"given":"Danielle","family":"Navarro","sequence":"first","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T14:32:57Z","timestamp":1600871577000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/wxn58"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,23]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31234\/osf.io\/wxn58","relation":{},"published":{"date-parts":[[2020,9,23]]},"subtype":"preprint"}' recorded_at: Wed, 05 Jul 2023 05:27:46 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1016%2Fj.tics.2019.07.009/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:46 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="text/xml"; rel="item", ; version="vor"; type="text/plain"; rel="item", ; version="tdm"; rel="license" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T14:30:24Z","timestamp":1688049024247},"reference-count":14,"publisher":"Elsevier BV","issue":"10","license":[{"start":{"date-parts":[[2019,10,1]],"date-time":"2019-10-01T00:00:00Z","timestamp":1569888000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.elsevier.com\/tdm\/userlicense\/1.0\/"}],"funder":[{"DOI":"10.13039\/501100011730","name":"Templeton World Charity Foundation","doi-asserted-by":"publisher"},{"DOI":"10.13039\/501100006188","name":"Einstein Stiftung Berlin","doi-asserted-by":"publisher"},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher"},{"DOI":"10.13039\/100000925","name":"John Templeton Foundation","doi-asserted-by":"publisher"},{"DOI":"10.13039\/501100013437","name":"Templeton Religion Trust","doi-asserted-by":"publisher"}],"content-domain":{"domain":["clinicalkey.jp","clinicalkey.com","cell.com","clinicalkey.es","clinicalkey.fr","clinicalkey.com.au","bmn.com","elsevier.com","sciencedirect.com"],"crossmark-restriction":true},"published-print":{"date-parts":[[2019,10]]},"DOI":"10.1016\/j.tics.2019.07.009","type":"journal-article","created":{"date-parts":[[2019,8,14]],"date-time":"2019-08-14T11:45:19Z","timestamp":1565783119000},"page":"815-818","update-policy":"http:\/\/dx.doi.org\/10.1016\/elsevier_cm_policy","source":"Crossref","is-referenced-by-count":121,"title":"Preregistration Is Hard, And Worthwhile","prefix":"10.1016","volume":"23","author":[{"given":"Brian A.","family":"Nosek","sequence":"first","affiliation":[]},{"given":"Emorie D.","family":"Beck","sequence":"additional","affiliation":[]},{"given":"Lorne","family":"Campbell","sequence":"additional","affiliation":[]},{"given":"Jessica K.","family":"Flake","sequence":"additional","affiliation":[]},{"given":"Tom E.","family":"Hardwicke","sequence":"additional","affiliation":[]},{"given":"David T.","family":"Mellor","sequence":"additional","affiliation":[]},{"given":"Anna E.","family":"van \u2019t Veer","sequence":"additional","affiliation":[]},{"given":"Simine","family":"Vazire","sequence":"additional","affiliation":[]}],"member":"78","reference":[{"key":"10.1016\/j.tics.2019.07.009_bb0005","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.1708274114","article-title":"The preregistration revolution","volume":"115","author":"Nosek","year":"2018","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"10.1016\/j.tics.2019.07.009_bb0010","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.jesp.2016.03.004","article-title":"Pre-registration in social psychology \u2013 a discussion and suggested template","volume":"67","author":"van \u2019t Veer","year":"2016","journal-title":"J. Exp. Soc. Psychol."},{"key":"10.1016\/j.tics.2019.07.009_bb0015","doi-asserted-by":"crossref","first-page":"632","DOI":"10.1177\/1745691612463078","article-title":"An agenda for purely confirmatory research","volume":"7","author":"Wagenmakers","year":"2012","journal-title":"Perspect. Psychol. Sci."},{"key":"10.1016\/j.tics.2019.07.009_bb0020","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1177\/0956797611430953","article-title":"Measuring the prevalence of questionable research practices with incentives for truth telling","volume":"23","author":"John","year":"2012","journal-title":"Psychol. Sci."},{"key":"10.1016\/j.tics.2019.07.009_bb0025","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1207\/s15327957pspr0203_4","article-title":"HARKing: hypothesizing after the results are known","volume":"2","author":"Kerr","year":"1998","journal-title":"Personal. Soc. Psychol. Rev."},{"key":"10.1016\/j.tics.2019.07.009_bb0030","article-title":"Preregistration becoming the norm in psychological science","author":"Nosek","year":"2018","journal-title":"APS Observer"},{"key":"10.1016\/j.tics.2019.07.009_bb0035","series-title":"The Compleat Academic: A Career Guide","first-page":"185","article-title":"Writing the empirical journal article","author":"Bem","year":"2004"},{"key":"10.1016\/j.tics.2019.07.009_bb0040","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1177\/1745691612459058","article-title":"Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability","volume":"7","author":"Nosek","year":"2012","journal-title":"Perspect. Psychol. Sci."},{"key":"10.1016\/j.tics.2019.07.009_bb0045","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1177\/2515245917747646","article-title":"Many analysts, one dataset: making transparent how variations in analytical choices affect results","volume":"1","author":"Silberzahn","year":"2018","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"10.1016\/j.tics.2019.07.009_bb0050","doi-asserted-by":"crossref","first-page":"1359","DOI":"10.1177\/0956797611417632","article-title":"False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant","volume":"22","author":"Simmons","year":"2011","journal-title":"Psychol. Sci."},{"key":"10.1016\/j.tics.2019.07.009_bb0055","article-title":"Ensuring the quality and specificity of preregistrations","author":"Veldkamp","year":"2018","journal-title":"PsyArXiv Preprints"},{"key":"10.1016\/j.tics.2019.07.009_bb0060","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1017\/S1049096516000810","article-title":"Standard operating procedures: a safety net for pre-analysis plans","volume":"49","author":"Lin","year":"2016","journal-title":"Polit. Sci. Polit."},{"key":"10.1016\/j.tics.2019.07.009_bb0065","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pbio.1002460","article-title":"Scientists\u2019 reputations are based on getting it right, not being right","volume":"14","author":"Ebersole","year":"2016","journal-title":"PLoS Biol."},{"key":"10.1016\/j.tics.2019.07.009_bb0070","doi-asserted-by":"crossref","first-page":"0021","DOI":"10.1038\/s41562-016-0021","article-title":"A manifesto for reproducible science","volume":"1","author":"Munaf\u00f2","year":"2017","journal-title":"Nat. Hum. Behav."}],"container-title":"Trends in Cognitive Sciences","original-title":[],"language":"en","link":[{"URL":"https:\/\/api.elsevier.com\/content\/article\/PII:S1364661319301846?httpAccept=text\/xml","content-type":"text\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.elsevier.com\/content\/article\/PII:S1364661319301846?httpAccept=text\/plain","content-type":"text\/plain","content-version":"vor","intended-application":"text-mining"}],"deposited":{"date-parts":[[2021,3,27]],"date-time":"2021-03-27T10:29:32Z","timestamp":1616840972000},"score":1,"resource":{"primary":{"URL":"https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S1364661319301846"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10]]},"references-count":14,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2019,10]]}},"alternative-id":["S1364661319301846"],"URL":"http:\/\/dx.doi.org\/10.1016\/j.tics.2019.07.009","relation":{"has-preprint":[{"id-type":"doi","id":"10.31234\/osf.io\/wu3vs","asserted-by":"object"}]},"ISSN":["1364-6613"],"subject":["Cognitive Neuroscience","Experimental and Cognitive Psychology","Neuropsychology and Physiological Psychology"],"container-title-short":"Trends in Cognitive Sciences","published":{"date-parts":[[2019,10]]},"assertion":[{"value":"Elsevier","name":"publisher","label":"This article is maintained by"},{"value":"Preregistration Is Hard, And Worthwhile","name":"articletitle","label":"Article Title"},{"value":"Trends in Cognitive Sciences","name":"journaltitle","label":"Journal Title"},{"value":"https:\/\/doi.org\/10.1016\/j.tics.2019.07.009","name":"articlelink","label":"CrossRef DOI link to publisher maintained version"},{"value":"simple-article","name":"content_type","label":"Content Type"},{"value":"\u00a9 2019 Elsevier Ltd. All rights reserved.","name":"copyright","label":"Copyright"}]}' recorded_at: Wed, 05 Jul 2023 05:27:46 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.1525%2Fcollabra.24091/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:47 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; version="vor"; type="application/pdf"; rel="item", ; version="vor"; rel="item", ; version="vor"; rel="license" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:28:09Z","timestamp":1687134489820},"reference-count":104,"publisher":"University of California Press","issue":"1","license":[{"start":{"date-parts":[[2021,1,1]],"date-time":"2021-01-01T00:00:00Z","timestamp":1609459200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"published-print":{"date-parts":[[2021,5,25]]},"abstract":"Exaggerated claims and low levels of reproducibility are commonplace in psychology and cognitive neuroscience, due to an incentive structure that demands \u201cnewsworthy\u201d results. My overall argument here is that in addition to methodological reform, greater modesty is required across all levels - from individual researchers to the systems that govern science (e.g., editors, reviewers, grant panels, hiring committees) - to redirect expectations regarding what psychological and brain science can effectively deliver. Empirical work and the reform agenda should pivot away from making big claims on narrow evidence bases or single tools and focus on the limitations of our individual efforts, as well as how we can work together to build ways of thinking that enable integration and synthesis across multiple modalities and levels of description. I outline why modesty matters for science including the reform agenda, provide some practical steps that we can take to embrace modesty, rebut common misconceptions of what modesty means for science, and present some limitations of the approach. Ultimately, by presenting a more sober view of our capacities and achievements, whilst placing work within a wider context that respects the complexity of the human brain, we will bolster the fidelity of scientific inference and thus help in a small way to generate a firmer footing upon which to build a cumulative science.<\/jats:p>","DOI":"10.1525\/collabra.24091","type":"journal-article","created":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T16:31:32Z","timestamp":1621960292000},"source":"Crossref","is-referenced-by-count":1,"title":"A Call for Greater Modesty in Psychology and Cognitive Neuroscience","prefix":"10.1525","volume":"7","author":[{"given":"Richard","family":"Ramsey","sequence":"first","affiliation":[{"name":"Department of Psychology, Macquarie University, Sydney, Australia"}]}],"member":"408","published-online":{"date-parts":[[2021,5,25]]},"reference":[{"key":"2021052516311117000_ref-59381"},{"key":"2021052516311117000_ref-59348"},{"key":"2021052516311117000_ref-59303"},{"key":"2021052516311117000_ref-59370"},{"key":"2021052516311117000_ref-59343"},{"key":"2021052516311117000_ref-59374"},{"key":"2021052516311117000_ref-59322"},{"key":"2021052516311117000_ref-59349"},{"key":"2021052516311117000_ref-59379"},{"key":"2021052516311117000_ref-59410"},{"key":"2021052516311117000_ref-59329"},{"key":"2021052516311117000_ref-59373"},{"key":"2021052516311117000_ref-59411"},{"key":"2021052516311117000_ref-59350"},{"key":"2021052516311117000_ref-59293"},{"key":"2021052516311117000_ref-59359"},{"key":"2021052516311117000_ref-59347"},{"key":"2021052516311117000_ref-59371"},{"key":"2021052516311117000_ref-59383"},{"key":"2021052516311117000_ref-59287"},{"key":"2021052516311117000_ref-59363"},{"key":"2021052516311117000_ref-59369"},{"key":"2021052516311117000_ref-59352"},{"key":"2021052516311117000_ref-59309"},{"key":"2021052516311117000_ref-59331"},{"key":"2021052516311117000_ref-59360"},{"key":"2021052516311117000_ref-59325"},{"key":"2021052516311117000_ref-59376"},{"key":"2021052516311117000_ref-59342"},{"key":"2021052516311117000_ref-59338"},{"key":"2021052516311117000_ref-59354"},{"key":"2021052516311117000_ref-59300"},{"key":"2021052516311117000_ref-59299"},{"key":"2021052516311117000_ref-59340"},{"key":"2021052516311117000_ref-59330"},{"key":"2021052516311117000_ref-59321"},{"key":"2021052516311117000_ref-59324"},{"key":"2021052516311117000_ref-59288"},{"key":"2021052516311117000_ref-59305"},{"key":"2021052516311117000_ref-59304"},{"key":"2021052516311117000_ref-59292"},{"key":"2021052516311117000_ref-59361"},{"key":"2021052516311117000_ref-59368"},{"key":"2021052516311117000_ref-59306"},{"key":"2021052516311117000_ref-59366"},{"key":"2021052516311117000_ref-59377"},{"key":"2021052516311117000_ref-59375"},{"key":"2021052516311117000_ref-59372"},{"key":"2021052516311117000_ref-59314"},{"key":"2021052516311117000_ref-59323"},{"key":"2021052516311117000_ref-59327"},{"key":"2021052516311117000_ref-59310"},{"key":"2021052516311117000_ref-59311"},{"key":"2021052516311117000_ref-59302"},{"key":"2021052516311117000_ref-59344"},{"key":"2021052516311117000_ref-59382"},{"key":"2021052516311117000_ref-59364"},{"key":"2021052516311117000_ref-59320"},{"key":"2021052516311117000_ref-59367"},{"key":"2021052516311117000_ref-59294"},{"key":"2021052516311117000_ref-59335"},{"key":"2021052516311117000_ref-59332"},{"key":"2021052516311117000_ref-59333"},{"key":"2021052516311117000_ref-59387"},{"key":"2021052516311117000_ref-59297"},{"key":"2021052516311117000_ref-59353"},{"key":"2021052516311117000_ref-59318"},{"key":"2021052516311117000_ref-59357"},{"key":"2021052516311117000_ref-59351"},{"key":"2021052516311117000_ref-59291"},{"key":"2021052516311117000_ref-59334"},{"key":"2021052516311117000_ref-59295"},{"key":"2021052516311117000_ref-59345"},{"key":"2021052516311117000_ref-59326"},{"key":"2021052516311117000_ref-59380"},{"key":"2021052516311117000_ref-59315"},{"key":"2021052516311117000_ref-59346"},{"key":"2021052516311117000_ref-59328"},{"key":"2021052516311117000_ref-59341"},{"key":"2021052516311117000_ref-59384"},{"key":"2021052516311117000_ref-59307"},{"key":"2021052516311117000_ref-59336"},{"key":"2021052516311117000_ref-59386"},{"key":"2021052516311117000_ref-59317"},{"key":"2021052516311117000_ref-59316"},{"key":"2021052516311117000_ref-59362"},{"key":"2021052516311117000_ref-59356"},{"key":"2021052516311117000_ref-59298"},{"key":"2021052516311117000_ref-59296"},{"key":"2021052516311117000_ref-59365"},{"key":"2021052516311117000_ref-59301"},{"key":"2021052516311117000_ref-59339"},{"key":"2021052516311117000_ref-59804"},{"key":"2021052516311117000_ref-59290"},{"key":"2021052516311117000_ref-59289"},{"key":"2021052516311117000_ref-59378"},{"key":"2021052516311117000_ref-59319"},{"key":"2021052516311117000_ref-59355"},{"key":"2021052516311117000_ref-59337"},{"key":"2021052516311117000_ref-59312"},{"key":"2021052516311117000_ref-59358"},{"key":"2021052516311117000_ref-59308"},{"key":"2021052516311117000_ref-59385"},{"key":"2021052516311117000_ref-59313"}],"container-title":"Collabra: Psychology","original-title":[],"language":"en","link":[{"URL":"http:\/\/online.ucpress.edu\/collabra\/article-pdf\/doi\/10.1525\/collabra.24091\/464786\/collabra_2021_7_1_24091.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/online.ucpress.edu\/collabra\/article-pdf\/doi\/10.1525\/collabra.24091\/464786\/collabra_2021_7_1_24091.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T16:31:32Z","timestamp":1621960292000},"score":1,"resource":{"primary":{"URL":"https:\/\/online.ucpress.edu\/collabra\/article\/7\/1\/24091\/117222\/A-Call-for-Greater-Modesty-in-Psychology-and"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":104,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,5,25]]}},"URL":"http:\/\/dx.doi.org\/10.1525\/collabra.24091","relation":{},"ISSN":["2474-7394"],"subject":["General Psychology"],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}' recorded_at: Wed, 05 Jul 2023 05:27:47 GMT - request: method: get uri: https://api.crossref.org/v1/works/10.31222%2Fosf.io%2Fh8xjw/transform body: encoding: UTF-8 string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close User-Agent: - http.rb/5.1.1 Host: - api.crossref.org response: status: code: 200 message: OK headers: Date: - Wed, 05 Jul 2023 05:27:47 GMT Content-Type: - application/vnd.citationstyles.csl+json Transfer-Encoding: - chunked Vary: - Accept Link: - ; rel="canonical", ; rel="license", ; title="Olmo Van den Akker"; rel="author", ; title="Marcel A. L. M. van Assen"; rel="author", ; title="Mahmoud Medhat Elsherif"; rel="author", ; title="Aline Claesen"; rel="author", ; title="Stefan Daniel Michel Gaillard"; rel="author", ; title="Thomas Rhys Evans"; rel="author", ; title="Max Korbmacher"; rel="author", ; title="Nihan Albayrak-Aydemir"; rel="author", ; title="Shilaan Alzahawi"; rel="author", ; title="Alexandra Sarafoglou"; rel="author", ; title="Filip Dechterenko"; rel="author", ; title="Sandra Grinschgl"; rel="author", ; title="Bradley James Baker"; rel="author", ; title="Jelte M. Wicherts"; rel="author" Access-Control-Expose-Headers: - Link Access-Control-Allow-Headers: - X-Requested-With, Accept, Accept-Encoding, Accept-Charset, Accept-Language, Accept-Ranges, Cache-Control Access-Control-Allow-Origin: - "*" Server: - Jetty(9.4.40.v20210413) X-Ratelimit-Limit: - '50' X-Ratelimit-Interval: - 1s X-Api-Pool: - public X-Rate-Limit-Limit: - '50' X-Rate-Limit-Interval: - 1s Permissions-Policy: - interest-cohort=() Connection: - close body: encoding: ASCII-8BIT string: '{"indexed":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:27:06Z","timestamp":1687134426973},"posted":{"date-parts":[[2023,5,10]]},"group-title":"MetaArXiv","reference-count":0,"publisher":"Center for Open Science","license":[{"start":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T00:00:00Z","timestamp":1683676800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"abstract":"

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lack methodological details and that undisclosed deviations from preregistered plans are frequent. Combining the strictness and consistency results highlights that biases due to researcher degrees of freedom are prevalent and likely in many preregistered studies. More comprehensive registration templates typically yielded stricter and hence better preregistrations. We did not find that effectiveness of preregistrations differed over time or between original and replication studies. Furthermore, we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrations and published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to write stricter preregistrations, adhere to these preregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.<\/p>","DOI":"10.31222\/osf.io\/h8xjw","type":"posted-content","created":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T05:02:40Z","timestamp":1683781360000},"source":"Crossref","is-referenced-by-count":1,"title":"The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency","prefix":"10.31222","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-0712-3746","authenticated-orcid":true,"given":"Olmo","family":"Van den Akker","sequence":"first","affiliation":[]},{"given":"Marjan","family":"Bakker","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7517-6081","authenticated-orcid":true,"given":"Marcel A. L. M.","family":"van Assen","sequence":"additional","affiliation":[]},{"given":"Charlotte Rebecca","family":"Pennington","sequence":"additional","affiliation":[]},{"given":"Leone","family":"Verweij","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-0540-3998","authenticated-orcid":true,"given":"Mahmoud Medhat","family":"Elsherif","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-0303-2441","authenticated-orcid":true,"given":"Aline","family":"Claesen","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/05f950310","id-type":"ROR","asserted-by":"publisher"}],"name":"KU Leuven"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-1956-7325","authenticated-orcid":true,"given":"Stefan Daniel Michel","family":"Gaillard","sequence":"additional","affiliation":[]},{"given":"Siu Kit","family":"Yeung","sequence":"additional","affiliation":[]},{"given":"Jan-Luca","family":"Frankenberger","sequence":"additional","affiliation":[]},{"given":"Kai","family":"Krautter","sequence":"additional","affiliation":[]},{"given":"Jamie Philip","family":"Cockcroft","sequence":"additional","affiliation":[]},{"given":"Katharina Sybille","family":"Kreuer","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-6670-0718","authenticated-orcid":true,"given":"Thomas Rhys","family":"Evans","sequence":"additional","affiliation":[]},{"given":"Fr\u00e9d\u00e9rique","family":"Heppel","sequence":"additional","affiliation":[]},{"given":"Sarah Fiona","family":"Schoch","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-8113-2560","authenticated-orcid":true,"given":"Max","family":"Korbmacher","sequence":"additional","affiliation":[]},{"given":"Yuki","family":"Yamada","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-3412-4311","authenticated-orcid":true,"given":"Nihan","family":"Albayrak-Aydemir","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-6892-4643","authenticated-orcid":true,"given":"Shilaan","family":"Alzahawi","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0031-685X","authenticated-orcid":true,"given":"Alexandra","family":"Sarafoglou","sequence":"additional","affiliation":[]},{"given":"Maksim","family":"Sitnikov","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0472-915X","authenticated-orcid":true,"given":"Filip","family":"Dechterenko","sequence":"additional","affiliation":[]},{"given":"Sophia","family":"Wingen","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0001-6666-9426","authenticated-orcid":true,"given":"Sandra","family":"Grinschgl","sequence":"additional","affiliation":[]},{"given":"Helena","family":"Hartmann","sequence":"additional","affiliation":[]},{"given":"Suzanne","family":"Stewart","sequence":"additional","affiliation":[]},{"given":"Catia Margarida","family":"Oliveira","sequence":"additional","affiliation":[]},{"given":"Sarah","family":"Ashcroft-Jones","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-1697-4198","authenticated-orcid":true,"given":"Bradley James","family":"Baker","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-2415-2933","authenticated-orcid":true,"given":"Jelte M.","family":"Wicherts","sequence":"additional","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T05:02:40Z","timestamp":1683781360000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/h8xjw"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,10]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31222\/osf.io\/h8xjw","relation":{},"published":{"date-parts":[[2023,5,10]]},"subtype":"preprint"}' recorded_at: Wed, 05 Jul 2023 05:27:47 GMT recorded_with: VCR 6.2.0