dpT | Asimov Stat Syst - rvizarreta/gundam-icarus GitHub Wiki

Frequentist coverage check for the GUNDAM fit. For each toy:

1. Throw the systematic nuisance parameters from the prior covariance.
2. Apply the systematic shift to the model prediction.
3. Add Poisson stat fluctuation to each reco bin to produce the toy's "data."
4. Run a Migrad fit on that data.
5. Record the best-fit LLH and other fit diagnostics.

Repeat 1000 times. Compare the LLH distribution to $\chi^2_k$ with $k = N_{\rm bins} - N_{\rm free}$ — if they match, the fit's uncertainty calculation has correct frequentist coverage.

Each toy = one gundamFitter invocation with --toy <seed>. Seeds are independent, so the ensemble is built by submitting $N$ jobs to the grid and post-processing per-toy ROOT files into summary tables.

On icarusgpvmXX, in a tmux session and the SL7 jobsub container:

tmux new -s toys
cd /exp/icarus/app/users/rvizarr/gundam-icarus
sh /exp/$(id -ng)/data/users/vito/podman/start_SL7dev_jsl.sh
source /cvmfs/icarus.opensciencegrid.org/products/icarus/setup_icarus.sh
kinit
htgettoken -a htvaultprod.fnal.gov --vaulttokenttl=1d --vaulttokenminttl=12h -i icarus

• grid_executable.sh — wrapper that runs one toy on a worker node. Rewrites the dataSetList input path to a local copy on the worker so the repo configs stay clean.
• Submit_GUNDAM.py — builds the jobsub_submit command; toggle DoSUBMIT to fire.

The fitter config used for this study (configs/RunConfigs/dpT/Asimov_StatSyst/config_Fitter_FakeData_dpT.yaml) must have these three flags set inside likelihoodInterfaceConfig:

likelihoodInterfaceConfig:
  enableStatThrowInToys: true
  enableEventMcThrow: false
  throwAsimovFitParameters: true

What each one does:

Without all three set correctly, the ensemble does not test what you think it tests.

Worker nodes can't see /exp/icarus/..., so GUNDAM, configs, and input ROOT files must all live on /pnfs.

cd /exp/icarus/app/users/rvizarr/gundam-icarus

tar cf ~/local_gundam.tar -C GUNDAM-1.9.X/Install .
tar cf ~/local_config.tar -C configs .

ifdh rm /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/tars/local_gundam.tar 2>/dev/null
ifdh rm /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/tars/local_config.tar 2>/dev/null
ifdh cp ~/local_gundam.tar /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/tars/local_gundam.tar
ifdh cp ~/local_config.tar /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/tars/local_config.tar

mkdir -p /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/inputs
ifdh cp /exp/icarus/data/users/rvizarr/medulla/icarus_numi_numu_mc_onbeam_offbeam_syst_gundam_withPCA.root \
        /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/inputs/icarus_numi_numu_mc_onbeam_offbeam_syst_gundam_withPCA.root

Verify all three landed:

ifdh ls /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/tars/
ifdh ls /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/inputs/

The repo dataSetListConfig_Asimov.yaml references the input file by /exp/icarus/... path. That works on the GPVM but not on a worker. Rather than maintain two versions of the config, grid_executable.sh runs ifdh cp to bring the input file to the worker's local scratch, then sed rewrites the untarred config on the worker to point at the local path. The repo stays untouched.

In Submit_GUNDAM.py, set NToys = 1 and DoSUBMIT = False to preview:

python3 Submit_GUNDAM.py

Confirm the printed command looks right. Then flip DoSUBMIT = True and submit:

python3 Submit_GUNDAM.py

Monitor:

jobsub_q -G icarus --user rvizarr

Once the job leaves the queue, verify the single-toy LLH:

source toys_env/bin/activate
python3 -c "
import uproot
a = uproot.open('/pnfs/icarus/scratch/users/rvizarr/toy_ensemble/output/output_Fitter_Toy_1.root')['FitterEngine/postFit/bestFitStats'].arrays(library='np')
print(f'llh_total = {float(a[\"totalLikelihoodAtBestFit\"][0]):.2f}')
print(f'llh_stat  = {float(a[\"statLikelihoodAtBestFit\"][0]):.2f}')
print(f'llh_pen   = {float(a[\"penaltyLikelihoodAtBestFit\"][0]):.2f}')
"

For $\chi^2_{19}$ (mean 19, std 6.2), a single value in the 5–35 range is plausible. If you get ~5 or below, the syst throws aren't happening; check the flag setup.

In Submit_GUNDAM.py: NToys = 1000, DoSUBMIT = True.

python3 Submit_GUNDAM.py

A few hours of wall time. Re-running is safe — if some jobs failed, resubmit only the missing seeds by setting the appropriate --offset and a smaller NToys.

Count outputs and check the queue periodically:

ifdh ls /pnfs/icarus/scratch/users/rvizarr/toy_ensemble/output/ | grep -c "output_Fitter_Toy_"
jobsub_q -G icarus --user rvizarr | tail -1

Two extraction scripts, both run inside the venv on the GPVM.

For the $\chi^2$ comparison plot. Produces one row per toy with llh_total, llh_stat, llh_penalty, etc.

See Notebooks/extract_toy_summary.py (or run the inline snippet — same idea).

For the Lane-style spread plots. Produces one row per toy with the bin contents of the Data_TH1D from both samples.

python3 extract_toy_spectra.py

Both CSVs are small (<1 MB) and scp easily to the laptop.

Two plotting scripts, both designed for Jupyter:

• plot_toy_ensemble.py — LLH${\rm total}$ histogram with $\chi^2{19}$ overlay.
• plot_mock_data_spread.py — Lane-style mock-data spread plots for signal and sideband samples.

Plots are saved to gundam-icarus/Notebooks/Plots/dpT/Asimov_StatSyst/.

If the study is set up correctly:

The mock-data spread plots should show the toy mean tracking the pre-fit CV across all bins of both samples.

/pnfs/icarus/scratch/users/rvizarr/toy_ensemble/
├── tars/
│   ├── local_gundam.tar
│   └── local_config.tar
├── inputs/
│   └── icarus_numi_numu_mc_onbeam_offbeam_syst_gundam_withPCA.root
└── output/
    └── output_Fitter_Toy_<N>.root