--- http_interactions: - request: method: get uri: https://api.rogue-scholar.org/posts/2b105b29-acbc-4eae-9ff1-368803f36a4d body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - api.rogue-scholar.org User-Agent: - http.rb/5.1.1 response: status: code: 200 message: OK headers: Content-Type: - application/json Content-Length: - '3338' Date: - Thu, 05 Oct 2023 20:37:20 GMT Server: - Fly/e440b950 (2023-09-20) Via: - 1.1 fly.io Fly-Request-Id: - 01HC0RY9B2FEZRK8X25XGRDVTW-fra body: encoding: UTF-8 string: '{"archive_url":null,"authors":[{"name":"Mark Rubin","url":null}],"blog":{"api":false,"archive_prefix":null,"authors":null,"backlog":0,"canonical_url":null,"category":"socialSciences","created_at":"2023-04-20","current_feed_url":null,"description":"Critical perspectives in metascience and the philosophy of science relating to open science, science reform, and the replication crisis.","favicon":"https://substackcdn.com/image/fetch/w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00187507-ddc5-4baa-9dae-79f9487d8bce_325x325.png","feed_format":"application/rss+xml","feed_url":"https://markrubin.substack.com/feed","filter":null,"funding":null,"generator":"Substack","home_page_url":"https://markrubin.substack.com","id":"ak4s224","issn":null,"language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","modified_at":"2023-09-09T22:05:00+00:00","plan":"Starter","prefix":"10.59350","relative_url":null,"slug":"markrubin","status":"active","title":"Critical Metascience","use_api":null,"use_mastodon":false,"user_id":"04a73ca4-9e60-4000-a0a0-087a13275eba","version":"https://jsonfeed.org/version/1.1"},"blog_name":"Critical Metascience","blog_slug":"markrubin","doi":"https://doi.org/10.59350/bbcsr-r4b59","id":"2b105b29-acbc-4eae-9ff1-368803f36a4d","image":"https://substack-post-media.s3.amazonaws.com/public/images/195477de-91ce-4a76-bae8-b6334fb5bcc7_500x379.jpeg","indexed_at":1688982864,"language":"en","published_at":1686086520,"reference":[{"doi":"https://doi.org/10.3386/w27250","key":"ref1"},{"doi":"https://doi.org/10.1371/journal.pbio.3000937","key":"ref2"},{"doi":"https://doi.org/10.1098/rsos.211037","key":"ref3"},{"doi":"https://doi.org/10.7554/eLife.67995","key":"ref4"},{"doi":"https://doi.org/10.31234/osf.io/5vfq6","key":"ref5"},{"key":"ref6","url":"https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html"},{"key":"ref7","url":"https://psyarxiv.com/nj4es"},{"doi":"https://doi.org/10.1038/s41562-021-01203-8","key":"ref8"},{"doi":"https://doi.org/10.1177/0956797611430953","key":"ref9"},{"doi":"https://doi.org/10.1207/s15327965pli0102_1","key":"ref10"},{"doi":"https://doi.org/10.31234/osf.io/wxn58","key":"ref11"},{"doi":"https://doi.org/10.1016/j.tics.2019.07.009","key":"ref12"},{"doi":"https://doi.org/10.1525/collabra.24091","key":"ref13"},{"doi":"https://doi.org/10.1177%20/0956797611417632","key":"ref14"},{"doi":"https://doi.org/10.1177","key":"ref15"},{"doi":"https://doi.org/10.31222/osf.io/h8xjw","key":"ref16"},{"key":"ref17","url":"https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice"},{"key":"ref18","url":"https://osf.io/tqwgp"},{"key":"ref19","url":"https://en.wikipedia.org/wiki/there_are_unknown_unknowns"}],"relationships":[],"summary":"Abstract I discuss Van Drimmelen\u2019s (2023) Metascience2023 presentation on researchers\u2019 decision making during the research process.","tags":["Metascience","Researcher Degrees Of Freedom","Open Science","Preregistration","Researcher Discretion"],"title":"The Preregistration Prescriptiveness Trade-Off and Unknown Unknowns in Science","updated_at":1686086520,"url":"https://markrubin.substack.com/p/the-preregistration-prescriptiveness"} ' recorded_at: Thu, 05 Oct 2023 20:37:20 GMT - request: method: get uri: https://doi.org/10.3386/w27250 body: encoding: ASCII-8BIT string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Thu, 05 Oct 2023 20:37:20 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '205' Connection: - close Location: - https://api.crossref.org/v1/works/10.3386%2Fw27250/transform Vary: - Accept Expires: - Thu, 05 Oct 2023 21:09:35 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=LEEMd6HLCZNEzs0iyNGHGYCr5nPR7k3X3TUnxQg1LzJ%2FzeToxpnzTOOz8CrOhE0D3i5zo0%2BLwsBq2qrdtOqGfRFMppfyv438CHCse7rZLL6clteUY0wfJoEjZhEuRcv8MGQX21M%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 811876441d880e48-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |-
In light of the \u201creplication crisis,\u201d some advocate for stricter standards and greater transparency in research methods. These efforts push toward a data analysis approach called \u201cconfirmatory data analysis\u201d (CDA). However, some (e.g., Baumeister, 2016; Finkel, Eastwick, & Reis, 2017; Goldin-Meadow, 2016) argued emphasizing CDA may restrict creativity and discovery. These scholars argued (sometimes inadvertently) for greater freedom to pursue an approach called \u201cexploratory data analysis\u201d (EDA). Unfortunately, many of those who push against stricter CDA standards don\u2019t realize EDA even exists, or fail to understand the philosophy and proper tools for exploration. In this paper, the meaning, tools, and ethics associated with EDA, CDA, and a relatively unknown but important approach called \u201crough CDA\u201d are clarified. Important distinctions are developed between EDA\/rough CDA\/CDA and problematic analysis activities including p-hacking, HARKing, and data mining. CDA, EDA, p-hacking, HARKing, etc., are situated in a unified (graphical) framework that clarifies ethical boundaries associated with each. In short, the proper data analytic approach depends on (1) intentions, and (2) transparency. Most psychological research isn\u2019t at a maturity level to justify CDA, and researchers have historically utilized tools that are mismatched to their research agenda. In the conclusion, guidelines are presented about how these typologies can be integrated into a cumulative research program that can help psychology move beyond the replication crisis.<\/p>","DOI":"10.31234\/osf.io\/5vfq6","type":"posted-content","created":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T19:35:52Z","timestamp":1576697752000},"source":"Crossref","is-referenced-by-count":3,"title":"Moving Beyond the \"Replication Crisis\": Understanding the Exploratory\/Confirmatory Data Analysis Continuum","prefix":"10.31234","author":[{"given":"Dustin","family":"Fife","sequence":"first","affiliation":[]},{"given":"Joseph Lee","family":"Rodgers","sequence":"additional","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T17:11:24Z","timestamp":1615223484000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/5vfq6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,18]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31234\/osf.io\/5vfq6","relation":{},"published":{"date-parts":[[2019,12,18]]},"subtype":"preprint"}' recorded_at: Thu, 05 Oct 2023 20:37:25 GMT - request: method: head uri: https://latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - latimes.com User-Agent: - http.rb/5.1.1 response: status: code: 301 message: Moved Permanently headers: Server: - awselb/2.0 Date: - Thu, 05 Oct 2023 20:37:25 GMT Content-Type: - text/html Content-Length: - '134' Connection: - close Location: - https://www.latimes.com:443/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html body: encoding: ASCII-8BIT string: '' recorded_at: Thu, 05 Oct 2023 20:37:25 GMT - request: method: head uri: https://psyarxiv.com/nj4es body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - psyarxiv.com User-Agent: - http.rb/5.1.1 response: status: code: 200 message: OK headers: Date: - Thu, 05 Oct 2023 20:37:26 GMT Content-Type: - text/html Content-Length: - '7666' Connection: - close Vary: - Accept-Encoding - Accept-Encoding Last-Modified: - Fri, 15 Sep 2023 22:47:59 GMT Etag: - '"6504df1f-1df2"' Cache-Control: - no-cache, no-store, max-age=0, must-revalidate Expires: - Mon, 01 Jan 1990 00:00:00 GMT Pragma: - no-cache Accept-Ranges: - bytes Strict-Transport-Security: - max-age=15724800 body: encoding: ASCII-8BIT string: '' recorded_at: Thu, 05 Oct 2023 20:37:26 GMT - request: method: get uri: https://doi.org/10.1038/s41562-021-01203-8 body: encoding: ASCII-8BIT string: '' headers: Accept: - application/vnd.citationstyles.csl+json Connection: - close Host: - doi.org User-Agent: - http.rb/5.1.1 response: status: code: 302 message: Found headers: Date: - Thu, 05 Oct 2023 20:37:26 GMT Content-Type: - text/html;charset=utf-8 Content-Length: - '229' Connection: - close Location: - https://api.crossref.org/v1/works/10.1038%2Fs41562-021-01203-8/transform Vary: - Accept Expires: - Thu, 05 Oct 2023 21:09:35 GMT Permissions-Policy: - interest-cohort=(),browsing-topics=() Cf-Cache-Status: - DYNAMIC Report-To: - '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=TtQ7rqLUOduu7%2FJ2xBaBcLwSkfLDGPj5apEY1l0TuoJcy5do7cvqigazGJtfE3UU4rTLW2OUFQp%2Bxb5eOYrTq01aimzo%2FKXbvsaJHbLBqjwszI6kOqTHfXPTc45vwxslgofgk84%3D"}],"group":"cf-nel","max_age":604800}' Nel: - '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}' Strict-Transport-Security: - max-age=31536000; includeSubDomains; preload Server: - cloudflare Cf-Ray: - 81187667c8a5b724-AMS Alt-Svc: - h3=":443"; ma=86400 body: encoding: UTF-8 string: |-
This
is an archived version of a blog post on preregistration. The first half of
the post argues that there is not a strong justification for preregistration
as a tool to solve problems with statistical inference (p-hacking); the second
half argues that preregistration has a stronger justification as one tool
(among many) that can aid scientists in documenting our projects. [Note that
this archival version exists only because the blog itself no longer does,
and as the original has been cited multiple times there is value in ensuring
that some version of the blog post remains accessible.]<\/p>","DOI":"10.31234\/osf.io\/wxn58","type":"posted-content","created":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T06:34:16Z","timestamp":1600842856000},"source":"Crossref","is-referenced-by-count":7,"title":"Paths
in strange spaces: A comment on preregistration","prefix":"10.31234","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-7648-6578","authenticated-orcid":true,"given":"Danielle","family":"Navarro","sequence":"first","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2020,9,23]],"date-time":"2020-09-23T14:32:57Z","timestamp":1600871577000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/wxn58"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,23]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31234\/osf.io\/wxn58","relation":{},"published":{"date-parts":[[2020,9,23]]},"subtype":"preprint"}'
recorded_at: Thu, 05 Oct 2023 20:37:29 GMT
- request:
method: get
uri: https://doi.org/10.1016/j.tics.2019.07.009
body:
encoding: ASCII-8BIT
string: ''
headers:
Accept:
- application/vnd.citationstyles.csl+json
Connection:
- close
Host:
- doi.org
User-Agent:
- http.rb/5.1.1
response:
status:
code: 302
message: Found
headers:
Date:
- Thu, 05 Oct 2023 20:37:30 GMT
Content-Type:
- text/html;charset=utf-8
Content-Length:
- '229'
Connection:
- close
Location:
- https://api.crossref.org/v1/works/10.1016%2Fj.tics.2019.07.009/transform
Vary:
- Accept
Expires:
- Thu, 05 Oct 2023 20:54:23 GMT
Link:
- "
Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies. We found that preregistrations often lack methodological details and that undisclosed deviations from preregistered plans are frequent. Combining the strictness and consistency results highlights that biases due to researcher degrees of freedom are prevalent and likely in many preregistered studies. More comprehensive registration templates typically yielded stricter and hence better preregistrations. We did not find that effectiveness of preregistrations differed over time or between original and replication studies. Furthermore, we found that operationalizations of variables were generally more effectively preregistered than other study parts. Inconsistencies between preregistrations and published studies were mainly encountered for data collection procedures, statistical models, and exclusion criteria. Our results indicate that, to unlock the full potential of preregistration, researchers in psychology should aim to write stricter preregistrations, adhere to these preregistrations more faithfully, and more transparently report any deviations from the preregistrations. This could be facilitated by training and education to improve preregistration skills, as well as the development of more comprehensive templates.<\/p>","DOI":"10.31222\/osf.io\/h8xjw","type":"posted-content","created":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T05:02:40Z","timestamp":1683781360000},"source":"Crossref","is-referenced-by-count":1,"title":"The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency","prefix":"10.31222","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-0712-3746","authenticated-orcid":true,"given":"Olmo","family":"Van den Akker","sequence":"first","affiliation":[]},{"given":"Marjan","family":"Bakker","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7517-6081","authenticated-orcid":true,"given":"Marcel A. L. M.","family":"van Assen","sequence":"additional","affiliation":[]},{"given":"Charlotte Rebecca","family":"Pennington","sequence":"additional","affiliation":[]},{"given":"Leone","family":"Verweij","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-0540-3998","authenticated-orcid":true,"given":"Mahmoud Medhat","family":"Elsherif","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-0303-2441","authenticated-orcid":true,"given":"Aline","family":"Claesen","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/05f950310","id-type":"ROR","asserted-by":"publisher"}],"name":"KU Leuven"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-1956-7325","authenticated-orcid":true,"given":"Stefan Daniel Michel","family":"Gaillard","sequence":"additional","affiliation":[]},{"given":"Siu Kit","family":"Yeung","sequence":"additional","affiliation":[]},{"given":"Jan-Luca","family":"Frankenberger","sequence":"additional","affiliation":[]},{"given":"Kai","family":"Krautter","sequence":"additional","affiliation":[]},{"given":"Jamie Philip","family":"Cockcroft","sequence":"additional","affiliation":[]},{"given":"Katharina Sybille","family":"Kreuer","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-6670-0718","authenticated-orcid":true,"given":"Thomas Rhys","family":"Evans","sequence":"additional","affiliation":[]},{"given":"Fr\u00e9d\u00e9rique","family":"Heppel","sequence":"additional","affiliation":[]},{"given":"Sarah Fiona","family":"Schoch","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-8113-2560","authenticated-orcid":true,"given":"Max","family":"Korbmacher","sequence":"additional","affiliation":[]},{"given":"Yuki","family":"Yamada","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-3412-4311","authenticated-orcid":true,"given":"Nihan","family":"Albayrak-Aydemir","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-6892-4643","authenticated-orcid":true,"given":"Shilaan","family":"Alzahawi","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0031-685X","authenticated-orcid":true,"given":"Alexandra","family":"Sarafoglou","sequence":"additional","affiliation":[]},{"given":"Maksim","family":"Sitnikov","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-0472-915X","authenticated-orcid":true,"given":"Filip","family":"Dechterenko","sequence":"additional","affiliation":[]},{"given":"Sophia","family":"Wingen","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0001-6666-9426","authenticated-orcid":true,"given":"Sandra","family":"Grinschgl","sequence":"additional","affiliation":[]},{"given":"Helena","family":"Hartmann","sequence":"additional","affiliation":[]},{"given":"Suzanne","family":"Stewart","sequence":"additional","affiliation":[]},{"given":"Catia Margarida","family":"Oliveira","sequence":"additional","affiliation":[]},{"given":"Sarah","family":"Ashcroft-Jones","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0002-1697-4198","authenticated-orcid":true,"given":"Bradley James","family":"Baker","sequence":"additional","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-2415-2933","authenticated-orcid":true,"given":"Jelte M.","family":"Wicherts","sequence":"additional","affiliation":[]}],"member":"15934","container-title":[],"original-title":[],"deposited":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T05:02:40Z","timestamp":1683781360000},"score":1,"resource":{"primary":{"URL":"https:\/\/osf.io\/h8xjw"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,10]]},"references-count":0,"URL":"http:\/\/dx.doi.org\/10.31222\/osf.io\/h8xjw","relation":{},"published":{"date-parts":[[2023,5,10]]},"subtype":"preprint"}' recorded_at: Thu, 05 Oct 2023 20:37:32 GMT - request: method: head uri: https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - metascience.info User-Agent: - http.rb/5.1.1 response: status: code: 301 message: Moved Permanently headers: Date: - Thu, 05 Oct 2023 20:37:33 GMT Server: - Apache Wpo-Cache-Status: - not cached Wpo-Cache-Message: - The request method was not GET (HEAD) X-Redirect-By: - WordPress Upgrade: - h2,h2c Connection: - Upgrade Location: - https://metascience.info/events/researchers-decision-making-navigating-ambiguity-in-research-practice/ Cache-Control: - max-age=7200 Expires: - Thu, 05 Oct 2023 22:37:33 GMT Host-Header: - c2hhcmVkLmJsdWVob3N0LmNvbQ== Content-Type: - text/html; charset=UTF-8 body: encoding: UTF-8 string: '' recorded_at: Thu, 05 Oct 2023 20:37:33 GMT - request: method: head uri: https://osf.io/tqwgp body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - osf.io User-Agent: - http.rb/5.1.1 response: status: code: 308 message: Permanent Redirect headers: Date: - Thu, 05 Oct 2023 20:37:34 GMT Content-Type: - text/html; charset=utf-8 Content-Length: - '249' Connection: - close Location: - https://osf.io/tqwgp/ X-Frame-Options: - SAMEORIGIN X-Cache-Status: - MISS Cache-Control: - no-cache, no-store, max-age=0, must-revalidate Expires: - Mon, 01 Jan 1990 00:00:00 GMT Pragma: - no-cache Strict-Transport-Security: - max-age=15724800 body: encoding: UTF-8 string: '' recorded_at: Thu, 05 Oct 2023 20:37:34 GMT - request: method: head uri: https://en.wikipedia.org/wiki/there_are_unknown_unknowns body: encoding: ASCII-8BIT string: '' headers: Connection: - close Host: - en.wikipedia.org User-Agent: - http.rb/5.1.1 response: status: code: 301 message: Moved Permanently headers: Date: - Thu, 05 Oct 2023 20:37:34 GMT Server: - mw1397.eqiad.wmnet X-Content-Type-Options: - nosniff Vary: - Accept-Encoding,X-Forwarded-Proto,Cookie Last-Modified: - Thu, 05 Oct 2023 20:37:34 GMT Location: - https://en.wikipedia.org/wiki/There_are_unknown_unknowns Content-Length: - '0' Content-Type: - text/html; charset=UTF-8 Age: - '0' X-Cache: - cp3070 miss, cp3070 miss X-Cache-Status: - miss Server-Timing: - cache;desc="miss", host;desc="cp3070" Strict-Transport-Security: - max-age=106384710; includeSubDomains; preload Report-To: - '{ "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }' Nel: - '{ "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}' Set-Cookie: - GeoIP=DE:NW:M__nster:51.92:7.57:v4; Path=/; secure; Domain=.wikipedia.org - NetworkProbeLimit=0.001;Path=/;Secure;Max-Age=3600 - WMF-Last-Access-Global=05-Oct-2023;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Mon, 06 Nov 2023 12:00:00 GMT - WMF-Last-Access=05-Oct-2023;Path=/;HttpOnly;secure;Expires=Mon, 06 Nov 2023 12:00:00 GMT X-Client-Ip: - 2001:16b8:188b:8800:a14e:42c1:cda5:f35 Cache-Control: - private, s-maxage=0, max-age=0, must-revalidate Connection: - close body: encoding: UTF-8 string: '' recorded_at: Thu, 05 Oct 2023 20:37:34 GMT recorded_with: VCR 6.2.0