CDR Tools - PADME-Experiment/padme-cdr GitHub Wiki

The CDR packages includes several tools to execute stand-alone tasks.

VerifyRunsToRemove - Compare content of DAQ data servers with files stored at LNF or CNAF
VerifyRun - Compare data from a single run between two sites
TransferRun - Copy a run from one site to another using parallel streams

VerifyRunsToRemove

VerifyRunsToRemove compares the current content of the DAQ data servers (l1padme3, l1padme4, l0padme2, and l0padme6) with the files already stored on the LNF Tier2 disk storage or on the CNAF Tape Library and reports a list of run directories which can be safely removed from the data servers.

The tool syntax is:

[leonardi@padmeui cdr]$ ./VerifyRunsToRemove -h
Usage: ./VerifyRunsToRemove [-T data_type] [-D dst_site] [-y year] [-h]
-T data_type    Define type of data to check (DAQ,MM,TMM)
-D dst_site     Define site to verify (LNF,CNAF,CNAF2,KLOE)
Default: compare content of year 2025 on DAQ servers with CNAF
[leonardi@padmeui cdr]$ ./VerifyRunsToRemove -h
Usage: ./VerifyRunsToRemove [-D dst_site] [-y year] [-h]
Default: compare content of year 2019 on DAQ servers with CNAF

If no arguments are given, the script will get a full list of the rawdata files on the DAQ data servers and print a line for each run directory specifying if all the files in that directory are also present at CNAF or if there are differences: in this last case the run must NOT be deleted on the DAQ data servers.

[leonardi@tier2-ui004 cdr]$ ./VerifyRunsToRemove
=== Run run_0080056_20250508_182635 - Successful match between DAQ-DAQ(l1padme3) and DAQ-CNAF ===
=== Run run_0080092_20250510_195429 - Successful match between DAQ-DAQ(l1padme3) and DAQ-CNAF ===
=== Run run_0080161_20250521_055912 - Successful match between DAQ-DAQ(l1padme3) and DAQ-CNAF ===
=== Run run_0080162_20250523_172306 - Successful match between DAQ-DAQ(l1padme3) and DAQ-CNAF ===
...etc...etc...

The list of commands to compare all on-line data directories (DAQ,MM,TMM) with what is stored on the LNF Tier2 disk storage is

[leonardi@padmeui cdr]$ ./VerifyRunsToRemove -T DAQ -D LNF
[leonardi@padmeui cdr]$ ./VerifyRunsToRemove -T MM -D LNF
[leonardi@padmeui cdr]$ ./VerifyRunsToRemove -T TMM -D LNF

The same to verify what is stored on the CNAF tape library is

[leonardi@tier2-ui004 cdr]$ ./VerifyRunsToRemove -T DAQ -D CNAF
[leonardi@tier2-ui004 cdr]$ ./VerifyRunsToRemove -T MM -D CNAF
[leonardi@tier2-ui004 cdr]$ ./VerifyRunsToRemove -T TMM -D CNAF

Please note that, to improve performance, files are just checked for existence and size in the destination site.

In the future the -c option will enable the safer but (much much) slower adler32 checksum verification of the compared files. If you feel paranoid and there is plenty of time before the data servers fill up, you will have the possibility to enable this feature. However, note that an adler32 checksum verification is active during each step of the PadmeCDR file copying process, so in principle this check is redundant.

VerifyRun

VerifyRun checks if all the files of a given run matches between two sites.

The tool syntax is:

[leonardi@padmeui cdr]$ ./VerifyRun -h
VerifyRun -R run_name [-T data_type] [-S src_site] [-D dst_site] [-s src_dir] [-d dst_dir] [-Y year] [-c] [-v] [-h]
  -R run_name     Name of run to verify
  -T data_type    Type of data to verify
  -S src_site     Source site.
  -D dst_site     Destination site.
  -s src_dir      Path to data directory if source is LOCAL, name of data server if source is DAQ.
  -d dst_dir      Path to data directory if destination is LOCAL, name of data server if destination is DAQ.
  -Y year         Specify year of data taking. Default: year from run name
  -c              Enable checksum verification (very time consuming!)
  -v              Enable verbose mode (repeat to increase level)
  -h              Show this help message and exit
  Available data types:  ['DAQ', 'MM', 'TMM']
  Available sites:       ['LNF', 'LNF2', 'CNAF', 'CNAF2', 'KLOE', 'DAQ', 'LOCAL']
  Available DAQ servers: ['l1padme3', 'l1padme4', 'padmesrv2', 'l0padme2', '192.168.60.10']

The default data type is DAQ, the default source and destination are CNAF and LNF.

The -v option can be repeated multiple times to activate more detailed reports:

[leonardi@padmeui cdr]$ ./VerifyRun -R run_0030359_20200918_045417
=== Run run_0030359_20200918_045417 - Successful match between CNAF and LNF ===

[leonardi@padmeui cdr]$ ./VerifyRun -R run_0030359_20200918_045417 -v
=== VerifyRun run_0030359_20200918_045417 between CNAF and LNF ===
2020-09-23 10:11:51 - at CNAF          run run_0030359_20200918_045417 contains 5 files
2020-09-23 10:11:51 - at LNF           run run_0030359_20200918_045417 contains 5 files
=== Run run_0030359_20200918_045417 - Successful match between CNAF and LNF ===

[leonardi@padmeui cdr]$ ./VerifyRun -R run_0030359_20200918_045417 -v -v
=== VerifyRun run_0030359_20200918_045417 between CNAF and LNF ===
2020-09-23 10:12:24 - at CNAF          run run_0030359_20200918_045417 contains 5 files
2020-09-23 10:12:24 - at LNF           run run_0030359_20200918_045417 contains 5 files
2020-09-23 10:12:24 - Starting verification of run run_0030359_20200918_045417 (5 files) between CNAF and LNF
run_0030359_20200918_045417_lvl1_00_000.root - OK - size        442
run_0030359_20200918_045417_lvl1_01_000.root - OK - size        442
run_0030359_20200918_045417_lvl1_02_000.root - OK - size        442
run_0030359_20200918_045417_lvl1_03_000.root - OK - size        442
run_0030359_20200918_045417_lvl1_04_000.root - OK - size        442
=== Run run_0030359_20200918_045417 - Successful match between CNAF and LNF ===

If the -c option (enable checksum) is used, the program retrieves the checksum of each file from both sites: this can be very slow!

TransferRun

TransferRun can be used to transfer all the files of a given run from one site to another. The parallel package is used to parallelize the copy process and must therefore be installed on the CDR server.

The tool syntax is:

[leonardi@padmeui cdr]$ ./TransferRun -h
TransferRun -R run_name [-S src_site] [-D dst_site] [-s src_dir]  [-d dst_dir] [-j jobs] [-h]
  -R run_name     Name of run to recover
  -S src_site     Source site. Default: CNAF
  -D dst_site     Destination site. Default: LNF
  -s src_dir      Path to data directory if source is LOCAL, name of data server if source is DAQ.
  -d dst_dir      Path to data directory if destination is LOCAL, name of data server if destination is DAQ.
  -j jobs         Number of parallel jobs to use. Default: 20
  -h              Show this help message and exit
  Available sites:   ['LNF', 'LNF2', 'CNAF', 'CNAF2', 'LOCAL', 'DAQ', 'KLOE']
  Available DAQ servers: ['l1padme3', 'l1padme4']

Please note that:

The KLOE tape library is not supported as a source site as it requires special access procedures.
The program does not include an internal check that all files were successfully copied: when TransferRun finishes, it is recommended to use VerifyRun to verify if the two copies of the run are consistent. If not, the TransferRun command can be re-issued and it will transfer all missing files.