Callback demo - roland-sipos/rubberdaq GitHub Wiki

rubberdaq_test_mock_dlh

This test application demonstrates the data movement between a producer and consumer (LatencyBuffer inserter) entity in the Readout subsystem. The two path can be enabled with either --cb for callbacks and --ct for threads. Default values are set to simulate WIBEthernetTypeAdapter data streams.

rubberdaq_mock_dlh
Usage: rubberdaq_test_mock_dlh [OPTIONS]

Options:
  -h,--help                   Print this help message and exit
  -n INT                      Number of data streams in the test.
  --rate FLOAT                Rate of data producers. [kHz]
  --lb_numa_node INT          NUMA node for LBs to allocate on.
  -c UINT                     Capacity/size of latency buffer.
  --run_secs INT              How many seconds the test should run.
  --ct                        Consumer threads mode.
  --cb                        Consume callback mode.

To run the app with threads for 60 seconds with 10 producer streams, you can do:

taskset -c 0-15,32-47 rubberdaq_test_mock_dlh --run_secs 60 -n 10 --ct

To run the app with callbacks for 60 seconds with 10 producer streams, you can do:

taskset -c 0-15,32-47 rubberdaq_test_mock_dlh --run_secs 60 -n 10 --cb

Taskset is used to lock-on a dedicated NUMA node, as we don't have control on the IOManager based intermediate buffers' memory allocation.

CPU affinity before performance measurements

It's important, to lock-on threads for accurate measurements of CPU and memory metrics. From another terminal, when allocations happened, and the killswitch is launched, you want to use the readout-affinity.py script targeting this test app.

The killswitch messages looks like this, you want to pin after this message:

Flipping killswitch that will start the countdown...
Application will terminate in 60 seconds...

An example CPU pinning file for np02-srv-001 looks similar to this:

{
  "rubberdaq_test_mock_dlh": {
    "--run_secs": {
      "parent": "0-15,32-47",
      "threads": {
        "rte-worker-1": "1",
        "rte-worker-3": "3",
        "rte-worker-5": "5",
        "rte-worker-7": "7",
        "rte-worker-9": "9",
        "rte-worker-11": "11",

        "cleanup-0": "0,1,2,3,32,33,34,35",
        "cleanup-1": "0,1,2,3,32,33,34,35",
        "cleanup-2": "0,1,2,3,32,33,34,35",
        "cleanup-3": "0,1,2,3,32,33,34,35",
        "cleanup-4": "0,1,2,3,32,33,34,35",
        "cleanup-5": "0,1,2,3,32,33,34,35",
        "cleanup-6": "0,1,2,3,32,33,34,35",
        "cleanup-7": "0,1,2,3,32,33,34,35",
        "cleanup-8": "0,1,2,3,32,33,34,35",
        "cleanup-9": "0,1,2,3,32,33,34,35",

        "producer-0": "4,5,6,7,36,37,38,39",
        "producer-1": "4,5,6,7,36,37,38,39",
        "producer-2": "4,5,6,7,36,37,38,39",
        "producer-3": "4,5,6,7,36,37,38,39",
        "producer-4": "4,5,6,7,36,37,38,39",
        "producer-5": "4,5,6,7,36,37,38,39",
        "producer-6": "4,5,6,7,36,37,38,39",
        "producer-7": "4,5,6,7,36,37,38,39",
        "producer-8": "4,5,6,7,36,37,38,39",
        "producer-9": "4,5,6,7,36,37,38,39"
      }
    }
  }
}

Measurements

Choose an appropriately high value for --run_secs, so you can look into and collect the metrics from uProf, PCM, OpMon to generate reports of the footprint of the application.