Investigating automated dissemination failures - thoth-pub/thoth-dissemination GitHub Wiki

This page explains how the Project Manager can check the status of automated dissemination attempts, and take action on failures. More detailed information on automated dissemination can be found in the workflow document.

Examples in this document are based on the Internet Archive dissemination process. Processes for other target platforms function similarly.

Note that the IA process was originally named bulk_ia_upload rather than ia_bulk_disseminate. The screenshots throughout reflect this.

Checking runs

  • Go to the relevant Action page (see table) for the process to be checked (e.g. Internet Archive). The first entry in the list of "workflow runs" should be the most recent run.
  • If the process has not run on schedule, inform the development team.
  • If the process has run and is marked with a green tick, the full process completed successfully. No further action is required.
  • If the process has run and is marked with a red cross, part(s) of the process failed. Click on the entry to find out more.

Investigating failed runs

  • The process has multiple parts. Parts which succeeded are marked with a green tick, and parts which failed are marked with a red cross. Click on a failed part to find out more.

  • Clicking on a part shows the log of what happened when it was run. Points in the log where failures occurred are marked with a red cross. The last few lines of the log usually contain error messages explaining the failure.

Interpreting error messages

  • Errors may occur for a variety of reasons. The automated process may have been set up incorrectly, the Thoth record for the work to be uploaded may be incomplete, the target platform may be experiencing an outage, and so on. Error messages should describe the problem, giving hints as to how to resolve it.

  • Example: "Error uploading to [platform]: credentials may be incorrect"
    Meaning: The process failed to log in to the target dissemination platform, so was not allowed to upload.
    Action required: The Project Manager or Thoth development team need to check that the login details are present and correct in https://github.com/thoth-pub/thoth-dissemination/settings/secrets/actions.

  • Example: "No PDF Full Text URL found for Work"
    Meaning: The Thoth record for the work to be uploaded doesn't include a PDF Publication with a canonical Location containing a Full Text URL. The process needs this in order to retrieve a copy of the PDF of the work which it can then upload to the target platform.
    Action required: The Project Manager needs to inform the publisher of the work about the problem. The Thoth ID of the work is displayed in the log, and also in the name of the process part. If the publisher then corrects the Thoth record, the Project Manager can use the "Re-run jobs" option on the main page of the failed run to retry the upload of the individual work. They should then check whether it succeeds or fails as above.

  • Example: "Error uploading to Internet Archive: We encountered an internal error. Please try again."
    Meaning: The process didn't work as expected, seemingly due to a problem at the target platform's end.
    Action required: The problem may be temporary, so the Project Manager can try "Re-run jobs" as above and see if the retried upload succeeds. Otherwise, they should inform the Thoth development team, who may need to contact the target platform for help.

  • Example: "ssl.SSLEOFError: EOF occurred in violation of protocol"
    Meaning: This error sometimes occurs with SWORDv2 workflows (e.g. Cambridge University Library), and may indicate that the connection to the platform was lost before the upload attempt could complete.
    Action required: The problem may be temporary, so try "Re-run jobs", as above.

  • Example: "No record found for publisher [ID]: ID may be incorrect"
    Meaning: The ID may indeed be incorrect. However, this error often simply means that the Thoth GraphQL API was temporarily unavailable at the time that the process was run (so searching for the ID didn't find anything) (see related issue).
    Action required: Double-check that the ID given in the error message refers to an existing Thoth publisher (e.g. by visiting thoth.pub/admin/publisher/[ID]). If not, check whether the relevant repository variable needs to be corrected. If (or when) the ID is correct, it may be necessary to "Re-run jobs" as above.

  • Example: "Not a valid Crossref DOI prefix: [prefix]"
    Meaning: The DOI entered in Thoth for this work has a publisher prefix which doesn't match any publisher prefixes registered in Crossref.
    Action required: Check if the prefix is registered to a different DOI agency (e.g. DataCite), rather than to Crossref. If so, it's meaningless to submit DOI information for this work to Crossref. The Project Manager should add the Thoth ID for the work to CR_ENV_EXCEPTIONS, as mentioned in the main workflow document.

  • Example: "Uploaded content is unacceptable. - Syntax error detected in pdf data. You may be able to repair the pdf file with a repair tool, pdftk is one such tool."
    Meaning: This is an Internet Archive error caused by their validity checks on uploaded PDF files.
    Action required: The Project Manager needs to inform the publisher of the work about the problem, and explain that the work will not be able to be archived to Internet Archive unless they provide a "repaired" version of the PDF. If and when this is provided, inform the Thoth development team so that they can manually fix the upload; at present, it's not possible to resolve the issue by re-running the job.

  • If in doubt, ask the Thoth development team.