Setting up a fddaq‐v5.2.0 development area - DUNE-DAQ/daqconf GitHub Wiki

Instructions for setting up a Far Detector software area for v5.2.0 development based on a recent nightly build

07-Oct-2024 - 🔺 Work in progress! 🔺 Steps 1-8 in the first section have been verified to work. The Reference Information, and the remaining sections, will be (re)verified soon.

Reference information:

Here are the suggested steps:

  1. create a new software area based on the latest nightly build (see step 1.iv for the exact dbt-create command to use)

    1. The steps for this are based on the latest instructions for daq-buildtools

    2. As always, you should verify that your computer has access to /cvmfs/dunedaq.opensciencegrid.org

    3. If you are using one of the np04daq computers, and need to clone packages, add the following lines to your .gitconfig file (once you do this, there will be no need to activate the web proxy each time you log in, and this means that you won't forget to disable it...):

      [http]
        proxy = http://np04-web-proxy.cern.ch:3128
        sslVerify = false
      
    4. Here are the steps for creating the new software area:

      cd <directory_above_where_you_want_the_new_software_area>
      source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
      setup_dbt latest_v5
      dbt-create -n NFD_DEV_241007_A9 [work_dir_name]  # work_dir_name is optional
      cd <work_dir_name if you specified one, or NFD_DEV_241007_A9 otherwise>
      
    1. Please note that if you are following these instructions on a computer on which the DUNE-DAQ software has never been run before, there are several system packages that may need to be installed on that computer. These are mentioned in this script. To check whether a particular one is already installed, you can use a command like yum list libzstd and check whether the package is listed under Installed Packages.
  2. add any desired repositories to the /sourcecode area. Some examples are provided in this section.

    If you want to be able to modify the test-session configuration below, or if you are updating the appmodel schema you will need to clone the appmodel package. In order to run the unit tests mentioned below, you will need to clone the dfmodules package. To just run the integration tests or the test-session as defined in the release, you will not need to clone any packages.

    1. decide if you want the very latest code, or a more stable set of packages that has been verified to work.

      Run this command to select the very latest code

      export use_very_latest_dunedaq_code=1
      

      or this one to select a more stable set of packages.

      export use_recent_verified_dunedaq_code=1
      
    2. clone the repositories (the following block has some extra directory checking; it can all be copy/pasted into your shell window)

      # change directory to the "sourcecode" subdir, if possible and needed
      if [[ -d "sourcecode" ]]; then
          cd sourcecode
      fi
      # double-check that we're in the correct subdir
      current_subdir=`echo ${PWD} | xargs basename`
      if [[ "$current_subdir" != "sourcecode" ]]; then
          echo ""
          echo "*** Current working directory is not \"sourcecode\", skipping repo clones"
      else
          # finally, do the repo clone(s)
          # We always get appmodel so that we can look at the configurations
          # If you want to run the dfmodules unit tests, clone dfmodules as well
          # appmodel and dfmodules are used as examples
          git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
          git clone https://github.com/DUNE-DAQ/dfmodules.git -b develop
          git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop
          if [[ "$use_very_latest_dunedaq_code" == "" ]]; then
              cd appmodel; git checkout 8eec2c9197; cd ..
              cd dfmodules; git checkout 8678c63293; cd ..
              cd daqsystemtest; git checkout 42af9d7ffb3; cd ..
          fi
          cd ..
      fi
      
      
  3. setup the work area and build the software. NB: even if you haven't checked out any packages the dbt-build is necessary to install the rte script passed to the applications started by drunc

    source env.sh
    dbt-build -j 20
    dbt-workarea-env
    
  4. The appmodel repository contains a sample configuration for a small test system. It can be exercised using the following steps:

    # from your Linux shell command line...
    drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config
    
    # from within the drunc shell...
    # Note that it is best to use a different run number each time that you "start".
    boot
    conf
    start 101
    enable-triggers
    # wait for a few seconds
    disable-triggers
    drain-dataflow
    stop-trigger-sources
    stop
    scrap
    terminate
    exit
    
    # Or, you can run everything in one Linux shell command:
    drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-1x1-config boot wait 5 conf wait 3 start 101 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate
    
    # after you exit `drunc`, you should wait for ~30 seconds for controller processes to exit before starting another session
    
  5. dfmodules contains unit tests which have been updated to use OKS, they can be run with

    dbt-unittest-summary.sh
    
  6. Integration tests can be run straight from the release with

    pytest -s $DUNE_DAQ_RELEASE_SOURCE/daqsystemtest/integtest/minimal_system_quick_test.py
    
  7. If developing drunc or druncschema, after these are cloned run pip install in the corresponding sourcecode subdirectories. Run dbt-workarea-env in the root of the working directory.

  8. When you return to working with the software area after logging out, the steps that you'll need to redo are the following:

    cd <work_dir>
    source ./env.sh
    dbt-build  # if needed
    dbt-workarea-env  # if needed
    

Running the test-session example configuration from appmodel with drunc

Look here.

The test-session Example Configuration (And setting up your own)

  • The current example configuration is here: https://github.com/DUNE-DAQ/appmodel/blob/develop/test/config/test-session.data.xml.

  • It consists of several files in both appmodel/test/config (configuration-specific) and appmodel/config/appmodel (somewhat more generic)

    1. test-session.data.xml is the main entry point for this configuration; it includes the other data files and defines the high-level objects like the Session, the DetectorConfig, and some of the Service entries.
    2. hosts.data.xml defines the hosts and processing resources
    3. wiecconfs.data.xml defines the "front end" electronics configuration
    4. ru-segment.data.xml defines the readout applications (ru-01, ru-02, ru-03)
    5. df-segment.data.xml defines the dataflow applications (df-01, df-02, df-03 and dfo-01) and their module configurations except
    6. trigger-segment.data.xml defines the trigger applications (tc-maker-1, mlt and hsi-to-tc-app) and its module configurations
    7. data-store-params.data.xml defines the output file configuration for the dataflow apps
    8. fsm.data.xml defines the state machine and the supported transition commands
    9. connections.data.xml defines the Network and Queue connection rules used to generate appropriate endpoints in the SmartDaqApplications
    10. moduleconfs.data.xml contains DAQ Module configuration objects for readout, dataflow, and trigger
  • These files are placed in the install/appmodel/share/test/config directory upon build (via the daq_install() CMake command, based on their location in appmodel/test/config), where OKS can find them

  • To create a new configuration, we are currently limited to copying files from an existing configuration, but tools are in development to generate the readout map from an existing one.

  • All OKS configurations are loaded from files relative to the current working director and the paths listed in DUNEDAQ_DB_PATH. If you are preparing a configuration in a separate directory you may want to prepend it to the list in DUNEDAQ_DB_PATH. By default it is set by dbt to include the install directory of any packages you have in sourcecode that have a config or schema directory with .xml files, followed by the packages from the release.

  • You can test the SmartDaqApplication generation of modules and connections using

    listApps test-session test/config/test-session.data.xml
    generate_modules_test test-session <app_name> test/config/test-session.data.xml # <app_name> is one of the apps listed by listApps
    

Editing database files

  • The data and schema XML files can be edited manually or by running the graphical dbe editors dbe_main and schemaeditor. To enable use of the dbe editors, you must first run the command spack load dbe then dbe_main -f /path/to/file for editing data files or schemaeditor -f <path to file> for editing schema files.

NB: Spack loading the dbe package updates your environment in ways that may affect the running of other commands. It is recomemned to do this in a separate shell or window.

Running with CRP4 as example with real hardware

  1. Log into the EHN1 DAQ cluster.
  2. setup a work area with a fresh nightly (NFD_DEV_240701_A9++) as above
  3. Clone the EHN1 config repository and source its setup_db_path.sh script
  4. Start drunc as above but boot sessions/crp4-session.data.xml crp4-oks-session
  5. You will be able to look at monitoring metrics and errors using grafana (v5 dashboards).

Running listrev with drunc

  1. Setup your area:
    # optionally you can have listrev locally doing 
    cd sourcecode       # optional
    git clone https://github.com/DUNE-DAQ/listrev.git -b develop # optional
    cd -                # optional                
    dbt-workarea-env
    dbt-build
  2. Launch drunc-unified-shell as in the test-session above
  3. In the drunc shell, boot the lr-session
    boot config/lrSession.data.xml lr-session
    
  4. Start a run, wait a while and stop the run

Evaluating the listrev Run

  • grep Exiting log_*lr-session_listrev* Will show the reported statistics.
  • The example is targeted at 100 Hz, so the expected number of messages seen by ReversedListValidator should be at least 100 times the run duration. There should be three lists in each message (from the three generators), so it should report 300 times the run duration for the number of lists.
  • Messages are round-robined to the two reversers, so each should see 50run_duration messages and 150run_duration lists. They should have approximately equal values for the reported counters.
  • Generators should generate 100*run_duration lists and send all (or almost all) of them.

Useful DBT and Spack commands for software areas

  • dbt-info release # prints out the release type and name, and the base release name (version)
  • dbt-info package <dunedaq_package_name> # prints out the package version and commit hash used by the release
  • dbt-info sourcecode # prints out the branch names of source repos under sourcecode, and marks those with local changes with "*"
  • spack find --loaded -N <external_package_name>, e.g. spack find --loaded -N boost # prints out the version of the specified external package that is in use in the current software area
  • spack info fddaq # prints out the packages that are included in the fddaq bundle for the current software area
  • spack info dunedaq # prints out the packages that are included in the dunedaq (common) bundle for the current software area

Also see here.

Monitoring the system

When running with drunc, metrics reports appear in the info_*.json files that are produced, one for each application (e.g. info_df-01.json). We can collate these, grouped by metric name, using python -m opmonlib.info_file_collator info_*.json (default output file is opmon_collated.json).

It is also possible to monitor the system using a graphic interface.

Steps to enable and view TRACE debug messages

Here are suggested steps for enabling and viewing debug messages in the TRACE memory buffer:

  • set up your software area, if needed (e.g. cd <work_dir>; source ./dbt-env.sh ; dbt-workarea-env)

  • export TRACE_FILE=$DBT_AREA_ROOT/log/${USER}_dunedaq.trace

    • this tells TRACE which file on disk to use for its memory buffer, and in this way, enables TRACE in your shell environment and in subsequent runs of the system with nanorc.
  • edit your the OKS session that you are using to add the TRACE_FILE env var to the drunc environment:

    • the highlighted lines in the following snippet of a config.data.xml file should be added (with the appropriate value for TRACE_FILE for your software area)

      Session snippet: (expand this section to see the snippet)
      <obj class="Session" id="mdapp-basic-session">
       <attr name="use_connectivity_server" type="bool" val="1"/>
       <attr name="connectivity_service_interval_ms" type="u32" val="2000"/>
       <attr name="data_request_timeout_ms" type="u32" val="1000"/>
       <attr name="data_rate_slowdown_factor" type="u32" val="1"/>
       <rel name="environment">
        <ref class="Variable" id="session-env-session-name-0"/>
        <ref class="Variable" id="session-env-session-name-1"/>
        <ref class="Variable" id="session-env-ers-verb"/>
        <ref class="Variable" id="session-env-ers-info"/>
        <ref class="Variable" id="session-env-ers-warning"/>
        <ref class="Variable" id="session-env-ers-error"/>
        <ref class="Variable" id="session-env-ers-fatal"/>
        <ref class="Variable" id="session-env-connectivity-server"/>
        <ref class="Variable" id="session-env-connectivity-port"/>
        <ref class="Variable" id="daqapp-cli-configuration"/>
        <ref class="Variable" id="session-env-trace_file"/>   ### add a line like this ###
       </rel>
       <rel name="disabled">
       </rel>
       <rel name="segment" class="Segment" id="root-segment"/>
       <rel name="infrastructure_applications">
        <ref class="ConnectionService" id="local-connection-server"/>
        <ref class="OpMonService" id="local-opmon-application-service"/>
       </rel>
       <rel name="detector_configuration" class="DetectorConfig" id="dummy-detector"/>
      </obj>
      
      ...
      
      <obj class="Variable" id="session-env-ers-warning">
       <attr name="name" type="string" val="DUNEDAQ_ERS_WARNING"/>
       <attr name="value" type="string" val="erstrace,throttle,lstdout"/>
      </obj>
      
      <obj class="Variable" id="session-env-trace_file">  ### add these 4 lines, also ###
       <attr name="name" type="string" val="TRACE_FILE"/>
       <attr name="value" type="string" val="/home/nfs/biery/dunedaq/16SepFDv5Test_0858/log/biery_dunedaq.trace"/>
      </obj>
      
      <obj class="Variable" id="session-env-session-name-0">
       <attr name="name" type="string" val="DUNEDAQ_PARTITION"/>
       <attr name="value" type="string" val="mdapp-basic-session"/>
      </obj>
      
  • run the application using the nanorc commands described above

    • this populates the list of already-enabled TRACE levels so that you can view them in the next step
  • run tlvls

    • this command outputs a list of all the TRACE names that are currently known, and which levels are enabled for each name
    • TRACE names allow us to group related messages, and these names typically correspond to the name of the C++ source file
    • the bitmasks that are relevant for the TRACE memory buffer are the ones in the "maskM" column
  • enable levels with tonM -n <TRACE NAME> <level>

    • for example, tonM -n DataWriter DEBUG+5 (where "5" is the level that you see in the TLOG_DEBUG statement in the C++ code)
  • re-run tlvls to confirm that the expected level is now set

  • re-run the application

  • view the TRACE messages using tshow | tdelta -ct 1 | more

    • note that the messages are displayed in reverse time order

A couple of additional notes:

  • For debug statements in our code that look like TLOG_DEBUG(5) << "test, test";, we would enable the output of those messages using a shell command like tonM -n <TRACE_NAME> DEBUG+5. A couple of notes on this...
    • when we look at the output of the bitmasks with the tlvls command, bit #5 is going to be offset by the number of bits that TRACE and ERS reserve for ERROR, WARNING, INFO, etc. messages. At the moment, the offset appears to be 8, so the setting of bit "DEBUG+5" corresponds to setting bit #13.
    • when we view the messages with tshow, one of the columns in its output shows the level associated with the message (the column heading is abbreviated as "lvl"). Debug messages are prefaced with the letter "D", and they include the number that was specified in the C++ code. So, for our example of level 5, we would see "D05" in the tshow output for the "test, test" messages.
  • There are many other TRACE 'commands' that allow you to enable and disable messages. For example,
    • tonMg <level> enables the specified level for all TRACE names (the "g" means global in this context)
    • toffM -n <TRACE NAME> <level> disables the specified level for the specified TRACE name
    • toffMg <level> disables the specified level for all TRACE names
    • tlvlM -n <TRACE name> <mask> explicitly sets (and un-sets) the levels specified in the bitmask
⚠️ **GitHub.com Fallback** ⚠️