Solving Pipeline Problems example (for admins) - ubtue/ub_tools GitHub Wiki

Sometimes there are problems within the pipeline or import that admins should be able to fix. The easiest case are FTP-related problems, e.g. connection problems or problems with incomplete exports on the BSZ side. In this case you might get an error mail, e.g. like this:

03:00 fetch_marc_updates.py: Script error (script: fetch_marc_updates.py)! (from: sobek.ub.uni-tuebingen.de)
bsz_util.py.DownloadRemoteFiles: Skip downloading since apparently generation of the files on the FTP server is not complete!

In this case, you should check if the generation of the file on the FTP server has finished (if not, maybe we have to contact the BSZ in this case). To check this, you need the FTP access credentials if you don't have them. check the script:

vim /usr/local/ub_tools/cronjobs/fetch_marc_updates.py

In this case, you will find a comment there:

#!/bin/python3
# -*- coding: utf-8 -*-
#
# A tool for the automation of tarball downloads from the BSZ.
# Config files for this tool look like this:
# (in addition, see BSZ.conf and smtp_server.conf)

All our conf files are in /usr/local/var/lib/tuelib. If you search for the related conf there, you will find the credentials in cronjobs/BSZ.conf.

(You can use any FTP program like FileZilla or also the ftp program to check).

Depending on the system type, you have to navigate either to the ixtheo or krimdok folder. In this example related to KrimDok you can see that the latest file has finished being written around 3:54.

grafik

If you go to the corresponding server (in this case sobek) and use crontab -l, you can see when the scripts are scheduled: grafik

In the screenshot you can see the following line: 0 3 * * * cd "$BSZ_DATEN" && "$BIN/fetch_marc_updates.py" "$EMAIL" > "$LOG_DIR/fetch_marc_updates.log" 2>&1

So this means that the script started at 3:00 which matches the information from the email. Since the latest file has been finished at 3:54, it simply means that the data is ready now and we have to re-run all processes.

The following processes are the most important processes in the pipeline and belong together:

  • fetch_marc_updates.py: This downloads the new differential data from the FTP server
  • merge_differential_and_full_marc_updates.sh: This merges the differential data together with the existing data on our server from the previous run.
  • create_refterm_file.py: This analyses the complete merged data and generates temporary files we need for running the pipeline.
  • initiate_marc_pipeline.py: This starts the major process of augmenting the marc data as well as importing all data into the Solr index (can run multiple hours).

Since the error has occurred in the 1st of the 4 scripts, you have to re-run all of them.

Do do that, you have to execute the commands from the crontab one by one. Remember that you have to replace the variables with the actual values you can see in the crontab header. Since you also might not want to overwrite existing log files, it is recommended to add suffixes like e.g. ".log.extra".

A short example regarding the 1st script would be:

cd /usr/local/ub_tools/bsz_daten && /usr/local/bin/fetch_marc_updates.py [email protected] > /usr/local/var/log/tuefind/fetch_marc_updates.log.extra 2>&1

In addition, since you cannot see what's going on while running the program, it is also strongly recommended to open a separate shell window and tail the corresponding log file. Sometimes this is not the same that you wrote in the call, since e.g. initiate_marc_pipeline.py will call another script that writes its own log.

ls -lrt /usr/local/var/log/tuefind

and then e.g.

tail -f /usr/local/var/log/tuefind/krimdok_marc_pipeline.log

Of course, this was just a brief summary. Many unexpected things can happen during administration. If you run into other error messages, you might even have to take your time & analyze the code of the scripts, try to understand what's going on, and find a proper solution. There can be hundreds of different cases which cannot be fully described in this wiki, so you should also take your time to build up experience across many of our workflows over a longer period of time!