• With 'save_eigenmode.py', the amplitude and phase corrections derived from a snapshot of the Sun (16-bit) observation now include a *.antCals.npz file. The amplitude correction is saved as the array 'normCorr', and the phase correction saved as the array 'phiCorr'. Both are intended to be multiplied to the beamform matrix directly.
  Specifically, the 'phiCorr' saves the full complex response of each antenna (mainly the instrument delay, but also phase distortion due to a filter), phase only.
  The 'normCorr' $= abs(V) / auto * wt\_SEFD$, where V is the leading eigenvector, auto is the amplitude of each antenna, and wt_SEFD = 1/SEFD is the relative weighting according to the SEFD of each antenna. The component arrays are also saved in the .npz file.

• Similar to the new approach (ver. 240813 below), please copy the .antCals.npz to a subfolder in burstt15:/home/ubuntu/BFM/.

• With 'zcu216_writeBFM_log.py', a corresponding option '--cal3' will try to locate the .antCals.npz files from the specified subfolder in BFM/ and use the 'normCorr' and 'phiCorr' for calibration.

• Additionally, 'zcu216_writeBFM_log.py' now also accepts an option '--adjust'. This option looks up the latest spectra (of the FPGA being operated on) and compares the relative gain between antennas with that saved in the calibration. The difference is used to adjust the weighting between antennas. If the gain is lowered by more than a factor of 3 (i.e. the amplitude will need to be multiplied by more than a factor of 3 compared to the calibration), then this antenna is flagged.

• With 'second_cal64.py', similar 'normCorr' and 'phiCorr' arrays are now saved in as 'solution_2ndCal.npz' within the 'cal_YYYYmmddHHMMSS.check' output folder.
  On burstt5, the output folder is copied to '/burstt5/disk1/2nd_cal/'.

• in 'rudp64_wrbd_3/', the 'write_2nd_matrix_64_log.py' has been updated to accept '--dcal <DIR>'. The <DIR> is the full path to the directory containing the 'solution_2ndCal.npz' file generated from the previous step.

There are a few components to the updates:

'zcu216_100g_config2_log.py' has both a lower case '-c' option and an upper case '-C' option. The lower case '-c ' can take a full config file, identical to the original version. However, it can also take an abbreviated '-c <board_config>', where <board_config> contains only the [board] section of the full config. The [common] section should be specified by the upper case '-C <common_config>'. This change is introduced to improve the maintenance of the config files. Changing one <common_config> is enough.

The 'save_eigenmode.py' now saves the bandpass equalization and the instrumental delay correction into a single array (.npy file) in the .check output folder.

Note that 'save_eigenmode.py' now takes the geometric delay into account. Therefore the calibration data (16-bit data) does not have to be taken at transit time. Any time when the Sun is above the local horizon can be used. However, due to antenna attenuation, if the data were taken at a lower elevation, a longer integration time (number of packets used) is needed when running 'save_eigenmode.py'. Typically, for data taken between 10 am and 2 pm, 32768 packets (i.e. save_eigenmode.py -n 32768) are used to get the bandpass solution.

The .npy files have been renamed and copied to burstt9:~/rfsoc/python_zcu216/BFM/eigen_240719/

(rfsoc) [ubuntu@burstt9 python_zcu216]$ ls BFM/eigen_240719/
FUS_row01.fpga223.fpga0.20240719142121.bin.eigen.h5.antCal.npy  FUS_row09.fpga107.fpga0.20240719142915.bin.eigen.h5.antCal.npy
FUS_row02.fpga224.fpga1.20240719142121.bin.eigen.h5.antCal.npy  FUS_row10.fpga105.fpga1.20240719142915.bin.eigen.h5.antCal.npy
FUS_row03.fpga222.fpga2.20240719142121.bin.eigen.h5.antCal.npy  FUS_row11.fpga110.fpga2.20240719142915.bin.eigen.h5.antCal.npy
FUS_row04.fpga229.fpga3.20240719142121.bin.eigen.h5.antCal.npy  FUS_row12.fpga248.fpga3.20240719142915.bin.eigen.h5.antCal.npy
FUS_row05.fpga252.fpga0.20240719142454.bin.eigen.h5.antCal.npy  FUS_row13.fpga111.fpga0.20240719143215.bin.eigen.h5.antCal.npy
FUS_row06.fpga250.fpga1.20240719142454.bin.eigen.h5.antCal.npy  FUS_row14.fpga108.fpga1.20240719143215.bin.eigen.h5.antCal.npy
FUS_row07.fpga244.fpga2.20240719142454.bin.eigen.h5.antCal.npy  FUS_row15.fpga109.fpga2.20240719143215.bin.eigen.h5.antCal.npy
FUS_row08.fpga247.fpga3.20240719142454.bin.eigen.h5.antCal.npy  FUS_row16.fpga104.fpga3.20240719143215.bin.eigen.h5.antCal.npy

There is a new script 'zcu216_writeBFM_log.py'. It takes '-b <BFM_ID>', where <BFM_ID> is now the subfolder name that contains the .npy file (i.e. eigen_240719 shown above). It also '--beam0' and '--flag' similar to the 'eigen2bfm.py'. It additionally takes a '--theta_off' option to shift the beams on top of the 'beam0' setting. The result is written to FPGA directly.

Combining the new 'zcu216_100g_config2_log.py' and 'zcu216_writeBFM_log.py', this new script 'reload_fpga.py' can be used on an arbitrary number of boards specified on the command line. Please check the usage for an explanation. (or wait till this part is fleshed up later)

An example usage:

./reload_fpga.py --bc bf256 [-c fpga_configs/fushan_sw] -B board_bf256_v2 -b FUS_eigen_250719 [--site FUS] --rows all --op "--file_flag fushan_flags_250629.txt -q" [--beam0 -7.5] --no-reload

[default option] in the square bracket can be omitted unless you want to change it. (e.g. need to change these for Nantou and Green Island)

FUS_eigen_250719 contains the latest antenna calibration (.npy files)

fushan_flags_250629.txt is the known bad antennas that should be flagged

--rows all

all is currently only applicable in Fushan. In general, please use a quoted string to specify the rows to operate on, e.g. --rows '1 2 3 4' The rows are counted from 1 to 16 here, but differently in the 2nd BFM. Sorry about that. I will make it more consistent later.

--op '-q' means to skip zeroing out the 1st BFM and directly writes the new matrix. makes it a little faster.

--no-reload is important because we just want to change the BFM on the fly. if we reload the bitcode, then we will need to kill the hung-up bursttd256 processes and restart the baseband program. So, be sure to add this option.

Omit the '--no-reload' and we will reload the bitcode for FPGA(s).

Given a set of baseband data from bf64 (i.e. ring?.YYYYmmsddHHMMSS.bin), this script derives the delay correction needed for each FPGA. The main result (the delay correction) is shown in the terminal. However, several diagnostic plots are also generated and saved in 'cal_YYYYmmddHHMMSS.check/'.

Note that the geometric delay is also taken into account (similar to the save_eigenmode.py mentioned above). Therefore calibration (baseband) data can be taken any time the Sun is above the local horizon, not limited to the transit time.

To run this script, please specify the baseband data from the same snapshot (different 'orders' with the same filename timestamp). Since this baseband data is obtained with 1st beamform in the FPGA, the flux of the Sun is attenuated by the 1st beams. Please use the 'check_times_target_bf256.py' script to estimate the optimal time for taking the snapshot and use it for analysis.

Similar to the bf64 version, this version works for data from the bf256 system (i.e. Fushan).

still works. (to be added later)

• load the 16-bit bitcode (reload_fpga.py --bc cal300 or --bc cal400)
• record a 16-bit full-baseband data of the Sun

assuming data is recorded as <bin_file> = fpga0.yyyymmddHHMMSS.bin

• run 'save_eigenmode.py' on the 16-bit data

save_eigenmode.py <bin_file> -n 32768 [--aref N] [--flim 300 700] [--site SITE] [--rot Degree] [--array <config_file>]
the '--flim 300 700' would be needed if the 16-bit data is recorded for 300-700MHz (cal300)
output files are saved under <out_dir> = fpga0.yyyymmddHHMMSS.bin.eigen.h5.check/

• copy the antCal.npy file to BFM folder

the BFM folder is typically located at /home/ubuntu/rfsoc/python_zcu216/BFM
create a <bfm_ID> folder under BFM/. e.g. <bfm_ID> = GBD_eigen_260114
find the antCals.npz file from the <out_dir> folder and copy it to the <bfm_ID> folder
rename the file as BFM/<bfm_ID>/row01.fpgaXXX.yyyymmddHHMMSS.antCals.npz
XXX is the IP of the FPGA. for example, IP=192.168.40.214, then XXX=214
the file name should match this format: '<bfm_ID>/.fpgaXXX..antCals.npz'

• reload with the bf16 bitcode
• run 'zcu216_writeBFM_log.py' with options '--cal3 [--adjust]' to apply the bandpass calibration and write the matrix

zcu216_writeBFM_log.py XXX -b <bfm_ID> -t <bfm_type> [--flim 300 700] --cal3 [--adjust]
the <bfm_type> can be 'idt' for identity matrix (the default is 'pos' for a 1x16 beamform)
again the '--flim 300 700' is needed if the calibration antCal.npy was derived for 300-700MHz
the --cal3 uses the new antCals.npz calibration information
the --adjust will optionally auto-adjust weighting between antennas according to the latest FPGA spectrum.