CELEBI ultimate version - askap-craco/CELEBI GitHub Wiki

CELEBI ultimate version provides more flexibility, improved optimization of resources and better diagnostic outputs compared to the previous versions. This document is a guideline for using CELEBI ultimate version on the Ozstar/NT supercomputer to process CRAFT detected FRB data.

Configuration files

A set of configuration parameters need to be defined in specific config file for each FRB. The configuration parameters are mostly similar to the previous version. An example config file is kept at /fred/oz313/processing/configs/ultimate_template.config which includes the most important parameters that need to be set for individual FRBs.

The default values of other parameters are included from various files located at /fred/oz313/processing/configs/defaults/. Setting the value of a default parameter after the including the default file will override its default value.

A detailed description of all parameters will be included here later (hopefully)!

Utility scripts

There are utility scripts that exist (can be found in /fred/oz313/processing/configs/scripts/) to help determine the right config parameters.

  1. getcentralfreq.sh - Get central frequency of observation.

Running the pipeline

To run CELEBI, ssh to the OzStar/NT head node, cd to /fred/oz313/processing/, load the NextFlow module (ml nextflow) and use the following command:

nextflow [CELEBI_ultimate dir]/pipelines/main.nf --[option] -c [config file] -w [work dir] > [log file]

where --option is used to run specific processing steps (see the next section) as well as passing optional configuration parameters (e.g. --nopolcal, --noflag).

Running with a shell script

It is recommended to run CELEBI on Ozstar/NT with Karol Desnos's custom nextflow version to enable better logging.

The standard bash script to run CELEBI is in /fred/oz313/processing/celebi_scripts/run_celebi_ultimate.sh which uses Karol's custom nextflow.

Command to run the standard bash script (from the processing directory):

/fred/oz313/processing/celebi_scripts/run_celebi_ultimate.sh -t [option] -c [FRBname] -r(should you choose to resume)

and to run multiple steps include all options within '...', e.g.

/fred/oz313/processing/celebi_scripts/run_celebi_ultimate.sh -t 'fcal corrfrb corrfld' -c [FRBname] -r(should you choose to resume)

Using CELEBI's grass and psychic attacks (Extreme caution recommended!!)

CELEBI is a grass and psychic type pokemon and can perform both grass type and psychic attacks. CELEBI performs an attack if the keyword is specified as an option in the `-t' argument. Attacks cannot be combined with other attacks or other options. Using CELEBI's attacks is not advised for inexperienced trainers. Extreme caution is recommended while using psychic attacks; things may go completely out of control.

Grass type attacks:

magical_leaf : Equivalent to fcal corrpcal imgpcal beampcal calcpcal

leaf_storm : Equivalent to fcal corrfld imgfld corrpcal imgpcal

charge_beam : Equivalent to beamfrb plotfrb

hyper_beam : Equivalent to beampcal beamfrb calcpcal plotfrb

Psychic attacks:

confusion : Equivalent to mfimage mfpos

psychic : Equivalent to fcal corrpcal imgpcal beampcal calcpcal corrfld imgfld beamfrb plotfrb

Processing steps

The basic processing steps can be run individually provided that the required inputs are already available (which may be outputs of other steps). To run an individual step pass the associated keyword from the following list as a command line argument.

Individual steps

  1. Process fluxcal ( --fcal ): Correlate Flux cal and find calibration solutions and delays. Requires no previous steps.

  2. Correlate polcal ( --corrpcal ): Requires outputs from Step 1.

  3. Correlate field ( --corrfld ): Requires outputs from Step 1.

  4. Correlate finder bins ( --corrfrb ): Requires outputs from Step 1.

  5. Image fluxcal ( --imgfcal ): Requires outputs from Step 1.

  6. Image polcal ( --imgpcal ): Requires outputs from Steps 1 and 2.

  7. Image field ( --imgfld ): Requires outputs from Steps 1 and 3.

  8. Image finder bins ( --imgfrb ): Requires outputs from Steps 1 and 4.

  9. Get FRB position ( --getpos ): Find offset corrected FRB position. Requires outputs from Steps 7 and 8.

  10. Beamform polcal ( --beampcal ): Requires outputs from Steps 1 and 6.

  11. Calculate polcal solutions ( --calcpcal ): Requires outputs from Step 10.

  12. Beamform FRB ( --beamfrb ): Requires outputs from Steps 1 and 8 (if params.usepos=false; see later subsections).

  13. Plot FRB dynamic specs ( --plotfrb ): Requires outputs from Steps 11 (unless params.nopolcal=true) and 12.

  14. Estimate structure maximizing DM ( --shrine ): Requires outputs from Step 13.

  15. Matched-filter imaging ( --mfimage ): Requires outputs from Steps 1 and 13.

  16. FRB position from MF imaging ( --mfpos ): Requires outputs from Steps 7 and 15.

  17. Search for additional pulses and periodicity ( --searchpulse ): Experimental. Requires outputs from Step 13.

  18. Extract zoomed in voltage data around detected pulses ( --extractpulse ): Experimental. Requires outputs from Step 17.

Multiple steps can be run together if the required inputs are already present or generated as outputs of another step(s) in the current run (in which case the other step(s) will wait for the previous one(s) to finish).

Localization

All localization steps (1 through 9 in the list above) can be run together with a single option --localize. Please mind the 'z'!

Beamforming

All beamforming steps (10 through 13 in the list above) can be run together with a single option --gethtr. It requires outputs of the localization steps (unless params.usepos=true; see the next subsection).

Beamforming with CRACO localization

In some cases, specifically for CRACO localized FRBs, the sky position of the FRB is already known to sufficient accuracy for beamforming. In these scenarios, if desired, beamforming can be done using the known FRB position without running localization steps.

To beamform at known FRB position go through the following steps.

  1. Set params.ra_frb and params.dec_frb to the known FRB position.
  2. Set parmas.usepos=true which forces beamforming at the given position.
  3. Run with --fcal to generate fluxcal solutions and delays.
  4. Run beamforming steps as desired.