Reproducibility - fcrimins/fcrimins.github.io GitHub Wiki

Marginal Revolution: When you get right down to it, who really cares? (1/23/16)

• "potentially useful tool is post-publication peer review."
• "One of the biggest problems with “economics as a science” is that economists themselves cannot usually admit how irrelevant so much of the work — even the quality work — turns out to be. I’m all for worrying about reproducibility, transparency, and the like, but sometimes I feel those micro-debates distract our attention from this bigger and broader problem and indeed help to obscure that problem."

Academic signaling and the post-truth world (12/23/16)

• "The incentives for academic researchers seem like they encourage a large volume of well-publicized spurious results."
• "Which means there is an ever-increasing army of people out there trying to find and report interesting results. But there's no guarantee that the supply of interesting results is infinite."

Another idea

• Separate the collection of data from its analysis. Those who collect the data should hold back some of the data as 'test' data to confirm the analyses, just like they do in data science competitions. [2 possible companies here: Data Collection companies and Data Analysis companies, both of which are under the broad heading of Data Handling]

fMRI clusterf****** (12/22/16)

NIH Plan to Fix Social Science (12/22/16)

Gelman on the Replication Crisis and Social Media (10/19/16)

A Skeptical View of the NSF (8/25/16)

The 7 biggest problems facing science, according to 270 scientists (7/21/16)

Email: "Irreprodicibility crisis"

• Christie Aschwanden synthesises the ongoing fraud and irreproducibility crises in science. http://phenomena.nationalgeographic.com/2015/08/29/ive-got-your-missing-links-right-here-29-august-2015/
  • http://fivethirtyeight.com/features/science-isnt-broken/
• A massive study quantifies psychology’s reproducibility problem. By me at the Atlantic.
  • http://www.theatlantic.com/health/archive/2015/08/psychology-studies-reliability-reproducability-nosek/402466/
• http://www.theatlantic.com/notes/2015/09/sweeping-psychologys-problems-under-the-rug/403726/

Email: So much wrong w this article

• If You Can't Make Predictions, You're Still in a Crisis
• http://slatestarcodex.com/2015/09/05/if-you-cant-make-predictions-youre-still-in-a-crisis/

Email: "good article on reproducibility and statistical power"

• http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

Email: "Improve reproducibility through open notebook science"

• The problem of irreproducibility is one which covers the health sciences, as well as a variety of other sciences, and demands attention. https://www.causes.com/actions/1767172-sign-the-petition-to-united-states-patent-and-trademark-office-the-office-of-science-and-technology-policy (Lots of references at this link.)
• Here are 2 more links, which are also linked to from the above.
  • https://www.whitehouse.gov/administration/eop/ostp
  • https://www.whitehouse.gov/blog/2013/05/16/introducing-project-open-data
• Science Magazine on Reproducibility: http://www.sciencemag.org/content/343/6168/229.summary
• http://www.nature.com/ng/journal/v41/n2/fig_tab/ng.295_T3.html
• Generally, see the links at the bottom of https://www.causes.com/actions/1767172-sign-the-petition-to-united-states-patent-and-trademark-office-the-office-of-science-and-technology-policy
  • Too much success in recent experiments: http://www.genetics.org/content/198/2/449.abstract
  • When are results too good to be true: http://www.genetics.org/content/198/2/449.abstract
  • Unreliable research: Trouble at the lab: http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble
  • Improving Scientific Reproducibility in an Age of International Competition and Big Data: [cited link doesn't work]

Email: "sometimes i'm wrong, reproducibility"

• http://sometimesimwrong.typepad.com/

Email: "Extraordinary laboratories, reproducibility, scientific method, astronomy vs chemistry"

• Extraordinary Laboratories http://www.ribbonfarm.com/2015/08/18/extraordinary-laboratories/

Psychology

• Email: "Failure to replicate"
  • Op-Ed Contributor: Psychology Is Not in Crisis http://rss.nytimes.com/c/34625/f/640377/s/4976b1aa/sc/32/l/0L0Snytimes0N0C20A150C0A90C0A10Copinion0Cpsychology0Eis0Enot0Ein0Ecrisis0Bhtml0Dpartner0Frss0Gemc0Frss/story01.htm
• Email: "Psychology findings strength"
  • Many Psychology Findings Not as Strong as Claimed, Study Says http://rss.nytimes.com/c/34625/f/640377/s/4959c684/sc/32/l/0L0Snytimes0N0C20A150C0A80C280Cscience0Cmany0Esocial0Escience0Efindings0Enot0Eas0Estrong0Eas0Eclaimed0Estudy0Esays0Bhtml0Dpartner0Frss0Gemc0Frss/story01.htm
• Email: (no subject)
  • Study Can't Confirm Results of Many Psychology Experiments http://rss.nytimes.com/c/34625/f/640377/s/4959c683/sc/28/l/0L0Snytimes0N0Caponline0C20A150C0A80C270Cscience0Cap0Eus0Esci0Everifying0Escience0Bhtml0Dpartner0Frss0Gemc0Frss/story01.htm

Email: "Patent a research methodology"

• reproduciblity is a problem (see wikipedia and http://validation.scienceexchange.com/#/reproducibility-initiative and http://www.nature.com/news/journals-unite-for-reproducibility-1.16259)
• new research method is a way to signal that research was honest, both consciously and subconsciously

Email: "example patents and other patent info"

• Improving reproducibility specifically: http://www.google.com/patents/US7517977
• Another improving reproducibility specifically: http://www.google.com/patents/US20150017669

Email: "When search for 'reproducibility'"

• RetractionWatch does not come up
• Also "journals lawyer-up"

Email: "mentions reproducibility"

• http://davegiles.blogspot.com/2015/04/introductory-statistics-for-data-science.html