Prepare the Replication report - labordynamicsinstitute/replicability-training GitHub Wiki

Pre-publication

For more detailed and up-to-date instructions, reference https://labordynamicsinstitute.github.io/replicability-training-curriculum/aea-jira-workflow-a-guide.html.

You've already been keeping the REPLICATION.md updated as you went along? Then you're almost done!

  • Clean up the REPLICATION.md - it should be factual, objective, and not written in the first person.
  • Delete the lines in REPLICATION.md that say they should be deleted.

Guidance for filling out parts of your report

Code description

  • Identify all Figures, Tables, and any in-text numbers.
    • Create a list, mapping each of them to a particular program and line number within the program using the code-check-TEMPLATE.xlsx file.
    • Commit that list.
  • ONLY include shortcomings in your report. Example:

Table 5: could not identify code that produces Table

Replication steps

  • Provide details about your process of accessing the code and data.
  • Do NOT detail things like "Saved them on my Desktop".
  • DO describe actions that you did as per instructions ("Added a config.do").
  • DO describe any other actions you needed to do, providing some detail ("Needed to make changes in multiple programs").

Findings

  • Briefly describe whether you were able to successfully replicate the tables and figures.
  • Be efficient with how much space you use:

DO THIS:

- Tables 1-3: successfully replicated.

DO NOT DO THIS:

- Table 1: successfully replicated 
- Table 2: successfully replicated 
- Table 3: successfully replicated
  • If there is a discrepancy between the figure in the paper and the one you produce, then include a screenshot from the paper and the graph generated by the programs for the comparison.
  • If figures are constructed from Excel files, make a reasonable effort to re-create the figure (i.e. don't take too long). If you cannot re-create it, then note this in your report.

In-text numbers

  • List page and line number of in-text numbers.
  • If ambiguous, cite the surrounding text, i.e., "the rate fell to 52% of all jobs: verified"
  • Do not include in-text numbers that refer to a table or figure. Example: "As shown in Table 2, the rate fell to 52% of all jobs."

Post-publication replication