Setting up a FarDet DAQ software development area

Instructions for setting up a Far Detector software area for v5.8.0 development based on a recent nightly build

19-Sep-2026 - 🔺 Work in progress! 🔺 Steps 1-8 in the first section have been verified to work. The Reference Information, and the remaining sections, will be (re)verified soon.

Reference information:

DUNE DAQ workflow statuses:
- CI Dashboard
- - Nightly build
- - Nightly integration tests
- - Checks of various dbt commands
- - Unit tests and clang format checks for all packages
general development: software development workflow, DUNE DAQ Software Style Guide, Doxygen documentation, Repo managers, GitHub teams and repos
suggested Spack commands to learn about the characteristics of an existing software area are available here
an introduction to the "assets" system, which we use to store files that are not code, is here
testing: NP04 computer inventory
other: Working Group task lists, List of DUNE-DAQ GitHub teams and repos
Main grafana dashboard, Suggestions for setting up a proxy to see Grafana displays outside CERN
Tag Collector
OKS System Description, DBE Editor Documentation
Generating Configuration Diagrams

Here are the suggested steps:

create a new software area based on the latest nightly build (see step 1.v for the exact dbt-create command to use)
1. The steps for this are based on the latest instructions for daq-buildtools
2. As always, you should verify that your computer has access to /cvmfs/dunedaq.opensciencegrid.org
3. If you are using one of the np04daq computers, and need to interact with GitHub servers (e.g clone packages), add the following lines to your $HOME/.gitconfig file. (Once you do this, there will be no need to activate the web proxy each time you want to run a git command that talks to the GitHub servers, and this means that you won't forget to disable it before running drunc...):
```
[http]
  proxy = http://np04-web-proxy.cern.ch:3128
  sslVerify = false
```
4. If you are using one of the np04daq computers, and need to install python packages into your python virtual environment using pip install, add the following lines to your $HOME/.config/pip/pip.conf file. (Once you do this, there will be no need to activate the web proxy each time you want to install a python package, and this means that you won't forget to disable it before running drunc...):
```
[global]
  proxy = http://np04-web-proxy.cern.ch:3128
```
5. Here are the steps for creating the new software area:
```
cd <directory_above_where_you_want_the_new_software_area>
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260919_A9 [work_dir_name]  # work_dir_name is optional
cd <work_dir_name if you specified one, or NFD_DEV_260919_A9 otherwise>
```
6. Please note that if you are following these instructions on a computer on which the DUNE-DAQ software has never been run before, there are several system packages that may need to be installed on that computer. These are mentioned in this script. To check whether a particular one is already installed, you can use a command like yum list libzstd and check whether the package is listed under Installed Packages.

add any desired repositories to the /sourcecode area. Some examples are provided in this section.

decide if you want the very latest code, or a more stable set of packages that has been verified to work.

Run this command to select the very latest code
```
export use_very_latest_dunedaq_code=1
```
or this one to select a more stable set of packages.
```
export use_recent_verified_dunedaq_code=1
```

clone the repositories (the following block has some extra directory checking; it can all be copy/pasted into your shell window)

# change directory to the "sourcecode" subdir, if possible and needed
if [[ -d "sourcecode" ]]; then
    cd sourcecode
fi
# double-check that we're in the correct subdir
current_subdir=`echo ${PWD} | xargs basename`
if [[ "$current_subdir" != "sourcecode" ]]; then
    echo ""
    echo "*** Current working directory is not \"sourcecode\", skipping repo clones"
else
    # finally, do the repo clone(s)
    # We always get appmodel so that we can look at the configurations
    # If you want to run the dfmodules unit tests, clone dfmodules as well
    # appmodel and dfmodules are used as examples
    git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
    git clone https://github.com/DUNE-DAQ/dfmodules.git -b develop
    git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop
    if [[ "$use_very_latest_dunedaq_code" == "" ]]; then
        cd appmodel; git checkout 3bf31cc979; cd ..
        cd dfmodules; git checkout d009c35880; cd ..
        cd daqsystemtest; git checkout 94e8a7d13b; cd ..
    fi
    cd ..
fi

setup the work area and build the software. NB: even if you haven't checked out any packages the dbt-build is necessary to install the rte script passed to the applications started by drunc
```
source env.sh
dbt-build -j 20
dbt-workarea-env
```
1. If you clone a Python package into your software area, it is recommended that you do that in the pythoncode subdirectory. For example:
```
cd $DBT_AREA_ROOT
cd pythoncode
git clone https://github.com/DUNE-DAQ/drunc.git
pip install drunc/.  # remember to include the dot! (or at least the slash)
cd $DBT_AREA_ROOT      
dbt-workarea-env
```
If you are running software tests that are not part of official data taking, set the ConnectivityService and drunc root controller ports in your local copy of the example configurations to random available port numbers. This helps avoids conflicts between users running test systems on the same computer at the same time.
```
daqconf_set_connectivity_service_port local-1x1-config config/daqsystemtest/example-configs.data.xml
daqconf_set_rc_controller_port local-1x1-config config/daqsystemtest/example-configs.data.xml
```

The daqsystemtest repository contains a sample configuration for a small test system. It can be exercised using the following steps:

# from your Linux shell command line...
drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config ${USER}-local-test

# from within the drunc shell...
# Note that it is best to use a different run number each time that you "start".
boot --no-override-logs
conf
start --run-number 101
enable-triggers
# wait for a few seconds
disable-triggers
drain-dataflow
stop-trigger-sources
stop
scrap
terminate
exit

# Or, you can run everything in one Linux shell command:
drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config ${USER}-local-test boot --no-override-logs wait 5 conf wait 3 start --run-number 101 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

# after you exit `drunc`, you should wait for several seconds for controller processes to exit before starting another session

Unit tests from software packages that have been cloned into the software development area can be run with the following command:
```
dbt-build --unittest
```
If this command is run with only daqsystemtest in the software area, the results will be rather underwhelming because that package doesn't have any unit tests defined. Additional repositories can be added to the software area using commands like the following:
```
cd $DBT_AREA_ROOT/sourcecode
git clone https://github.com/DUNE-DAQ/dfmodules.git -b <version>
cd ..
dbt-build --unittest
```

Running integration tests

There are a substantial number of integration/regression tests available. They are located in the integtest directory of several repositories, including daqsystemtest. To run them, we can use the global bundle script (dunedaq_integtest_bundle.sh) or we can run them directly.

The global bundle script has the word "bundle" in its name because it can be used to run many integration tests with one command. It is referred to as the "global" bundle script because it provides access to the integration tests in all of the C++ repositories. It has options that allow users to select which tests are run and the conditions under which they are run. Please see this page for more information about this script. Here is a summary of the options that are available in this script as of 09-June-2026:

Usage:
dunedaq_integtest_bundle.sh [option(s)]

Options:
    -h, --help : prints out usage information
    -r <the list of repositories for which integtests will be run>
       - this can be the name of a single repo; it defaults to "daqsystemtest"
       - it can be a pipe-delimited string with a list of repos, e.g. 'dfmodules|trigger'
       - it can have the special value of "all" - integtests in all repos will be run
       - it can have the special value of "local" - integtests in locally-cloned repos will be run
    -R <the list of repositories to be excluded>
       - this can be the name of a single repo
       - it can be a pipe-delimited string with a list of repos, e.g. 'dfmodules|trigger'
    -k, --include <pipe-delimited string to select the tests that will be run ('egrep -i' match to test name)>
    -x, --exclude <pipe-delimited string to specify tests to be excluded ('egrep -i' match to test name)>
    --random-subset <count> : randomly picks the specified number of tests from the results of -r/-k/-x
    --list-only : list the tests that match the requested patterns without running them
    --verbosity <level> : requested level of console messages, in range 1-6, where 1 is least, 6 is DRUNC debug
    --stop-on-failure : causes the script to stop when one of the integtests reports a failure
    --tmpdir <dir> : specifies a root directory to use for test output, e.g. a directory instead of '/tmp'
    --trigger-full-rc-output <phrase that will trigger the full printout of run control messages>
       - the phrase can be a Python regex, which can be useful in handling colorized text
    --concise-output : suppresses run control and DAQApp messages in order to focus on test results
       - this is equivalent to "--verbosity 1", and this option may be removed at some point in time
    -n <number of times to run each individual test, default=1>
    -N <number of times to run the full set of selected tests, default=1>
    --pytest-options <options> : string with one or more dunedaq-specific command-line options to pass to Pytest
       - available options include the following:
         --dunerc-path <path> : Path to DUNE run control. Default is to search in $PATH
         --skip-resource-checks : Whether to skip the node resource (CPU/Memory) checks for this test
         --process-manager-type <type> : The run control process manager type to use for this test, e.g. ssh-standalone
         --dunerc-option <option-name> <option-value> : Repeatable, run control arguments without leading dashes
             for example, --dunerc-option log-level debug
       - example: --pytest-options "--skip-resource-checks --process-manager-type ssh-standalone --dunerc-option no-override-logs"

If we want to run an integration test directly, we can use commands like the following:

pytest -s ${DAQSYSTEMTEST_SHARE}/integtest/minimal_system_quick_test.py

cd $DBT_AREA_ROOT/sourcecode/daqsystemtest/integtest
pytest -s minimal_system_quick_test.py

In case of errors, log files from the bundle script and the DAQ processes can be found in the /tmp/pytest-of-${USER} directory. pytest creates a new subdirectory for each of our integration/regression tests, so you will see subdirs with names like pytest-1234. The application log files will be in "run" directories underneath those subdirs, e.g. runcurrent. A quick short-hand is cd /tmp/pytest-of-${USER}/pytest-current/runcurrent.

If developing drunc or druncschema, after these are cloned, run pip install in the corresponding pythoncode subdirectories. Run dbt-workarea-env in the root of the working directory.
When you return to working with the software area after logging out, the steps that you'll need to redo are the following:
```
cd <work_dir>
source ./env.sh
dbt-build  # if needed
dbt-workarea-env  # if needed
```

Useful DBT and Spack commands for software areas

dbt-info release # prints out the release type and name, and the base release name (version)
dbt-info package <dunedaq_package_name> # prints out the package version and commit hash used by the release
dbt-info sourcecode # prints out the branch names of source repos under sourcecode, and marks those with local changes with "*"
spack find --loaded -N <external_package_name>, e.g. spack find --loaded -N boost # prints out the version of the specified external package that is in use in the current software area
spack info fddaq # prints out the packages that are included in the fddaq bundle for the current software area
spack info coredaq # prints out the packages that are included in the coredaq (common) bundle for the current software area

Also see here.

Instructions for using the `HDF5LIBS_TestDumpRecord` utility

This utility can be used to print out information from the HDF5 raw data files. To invoke it use

HDF5LIBS_TestDumpRecord <filename>

Getting an overview of the HDF5 file structure

h5dump-shared -H <filename>

Dumping the binary content of a certain block from HDF5 file

This is another use of the h5dump-shared utility. This case uses the following command-line arguments:

the HDF5 path of the block we want to dump (-d )
the output binary file name (-o <output_file>)
the HDF5 file to be dumped

An example is:

h5dump-shared -d /TriggerRecord00001.0000/RawData/Detector_Readout_0x00000064_WIBEth -
bLE -o dataset1.bin test_raw_run001041_0000_df-01_dw_0_20241113T163255.hdf5

Once you have the binary file, you can examine it with tools like Linux od (octal dump), for example

od -x dataset1.bin

Monitoring the system

When running the DAQ system outside of CERN, metrics reports appear in the info_*.json files that are produced, one for each application (e.g. info_df-01.json). We can collate these, grouped by metric name, using info_file_collator info*.json (default output file is opmon_collated.json).

To summarize the metrics produced by a regression test, we need to cd to the pytest output directory first:
- cd /tmp/pytest-of-${USER}/pytest-current/runcurrent
- info_file_collator info*.json
- edit opmon_collated.json
Note that when we run the system multiple times from the same directory, the file-based metrics are added to existing files for each new run. So, it is possible to have metrics from different, widely-separated-in-time, runs in the same opmon files.

When running the system at EHN1, we typically monitor the system using a graphic interface.

Steps to enable and view TRACE debug messages

Here are suggested steps for enabling and viewing debug messages in the TRACE memory buffer:

set up your software area, if needed (e.g. cd <work_dir>; source ./env.sh ; dbt-workarea-env)
export TRACE_FILE=$DBT_AREA_ROOT/log/${USER}_dunedaq.trace
- this tells TRACE which file on disk to use for its memory buffer, and in this way, enables TRACE in your shell environment and in subsequent runs of the system with drunc.
modify the OKS session that you are using to add the TRACE_FILE env var to the drunc environment. There is a script available to do this. Here is an example of modifying the local-1x1-config OKS session in the daqsystemtest example configs:
- daqconf_set_session_env_var local-1x1-config config/daqsystemtest/example-configs.data.xml TRACE_FILE $TRACE_FILE
run a demo DAQ system using the drunc commands described above
- this populates the list of available TRACE names so that you can view them in the next step
run tlvls
- this command outputs a list of all the TRACE names that are currently known, and which levels are enabled for each name
- TRACE names allow us to group related messages, and these names typically correspond to the name of the C++ source file
- the bitmasks that are relevant for the TRACE memory buffer are the ones in the "maskM" column
enable levels with tonM -n <TRACE NAME> <level>
- for example, tonM -n DataWriterModule DEBUG+5 (where "5" is the level that you see in the TLOG_DEBUG statement in the C++ code)
re-run tlvls to confirm that the expected level is now set
- e.g. tlvls | grep DataWriter
re-run the demo DAQ system
view the TRACE messages using tshow | tdelta -ct 1 | more
- note that the messages are displayed in reverse time order
- e.g. tshow | tdelta -ct 1 | grep DataWriterModule | head

A couple of additional notes:

For debug statements in our code that look like TLOG_DEBUG(5) << "test, test";, we would enable the output of those messages using a shell command like tonM -n <TRACE_NAME> DEBUG+5. A couple of notes on this...
- when we look at the output of the bitmasks with the tlvls command, bit #5 is going to be offset by the number of bits that TRACE and ERS reserve for ERROR, WARNING, INFO, etc. messages. At the moment, the offset appears to be 8, so the setting of bit "DEBUG+5" corresponds to setting bit #13.
- when we view the messages with tshow, one of the columns in its output shows the level associated with the message (the column heading is abbreviated as "lvl"). Debug messages are prefaced with the letter "D", and they include the number that was specified in the C++ code. So, for our example of level 5, we would see "D05" in the tshow output for the "test, test" messages.
There are many other TRACE 'commands' that allow you to enable and disable messages. For example,
- tonMg <level> enables the specified level for all TRACE names (the "g" means global in this context)
- toffM -n <TRACE NAME> <level> disables the specified level for the specified TRACE name
- toffMg <level> disables the specified level for all TRACE names
- tlvlM -n <TRACE name> <mask> explicitly sets (and un-sets) the levels specified in the bitmask

Setting up a FarDet DAQ software development area - DUNE-DAQ/daqconf GitHub Wiki

Instructions for setting up a Far Detector software area for v5.8.0 development based on a recent nightly build

Useful DBT and Spack commands for software areas

Instructions for using the `HDF5LIBS_TestDumpRecord` utility

Getting an overview of the HDF5 file structure

Dumping the binary content of a certain block from HDF5 file

Monitoring the system

Steps to enable and view TRACE debug messages

⚠️ GitHub.com Fallback ⚠️

Setting up a FarDet DAQ software development area - DUNE-DAQ/daqconf GitHub Wiki

Instructions for setting up a Far Detector software area for v5.8.0 development based on a recent nightly build

Useful DBT and Spack commands for software areas

Instructions for using the HDF5LIBS_TestDumpRecord utility

Getting an overview of the HDF5 file structure

Dumping the binary content of a certain block from HDF5 file

Monitoring the system

Steps to enable and view TRACE debug messages

⚠️ **GitHub.com Fallback** ⚠️

Instructions for using the `HDF5LIBS_TestDumpRecord` utility

⚠️ GitHub.com Fallback ⚠️