The platform comprises three main components: Central, Network, and Distributed. However, a Builder component is used to build them. The components can be installed on a single computer or every component on a dedicated computer (We recommend this approach).

The components have the following requirements:

• Builder
  • Ubuntu 20.04 LTS or Alpine Linux 3.20
  • Git: 2.43.0 or newer
  • GNU bash: 5.2.21 or newer
  • Your favorite text editor
  • Docker: v20.10.22 or newer

📝 *Note: You can use https://github.com/pigovsky/useful-scripts/blob/main/docker/install-docker.sh script to install the newest docker, docker compose, and to add your user to the docker group.

• envsubst: 0.22.5 or newer
• Memory: 768 MB RAM or more
• Disk: 10 GB or more
• CPU: at least one core

• Central
  • Ubuntu 20.04 LTS or Alpine Linux 3.20
  • Git: 2.43.0 or newer
  • Clone the latest SigNoz repository.
    • If you are going to deploy SigNoz somewhere else or use its cloud version, then you can skip this dependency
  • Your favorite text editor to edit configuration files
  • Docker: v20.10.22 or newer
  • Docker Compose: v2.29.7 or newer
  • Memory: 768 MB RAM or more
  • Disk: 10 GB or more
  • CPU: at least one core

📝 Note: Refer to the SigNoz documentation for further requirements

• Network event manager

  • Ubuntu 20.04 LTS or Alpine Linux 3.20
  • Docker: v20.10.22 or newer
  • Docker Compose: v2.29.7 or newer
  • Memory: 4 GB RAM or more
  • Disk: 4 GB or more
  • CPU: at least 4 cores

• Distributed event manager

  • Ubuntu 20.04 LTS or Alpine Linux 3.20
  • Docker: v20.10.22 or newer
  • Docker Compose: v2.29.7 or newer
  • Memory: 768 MB RAM or more
  • Disk: 4 GB or more
  • CPU: at least 4 cores

Follow the sections below to deploy each component.

The Builder is used to build docker images of all other components and to configure them.

The deployment of the Builder requires following these steps

Navigate to some directory where you have writing access and would like to store the platform source files.

Clone the Repository and navigate to the project directory

git clone https://github.com/Smart-Edge-Lab/SeQaM.git

cd SeQaM

🪲 Common Issues:

1. Permission to run Docker: If you encounter issues running Docker (e.g., "permission denied"), it could be because the user doesn’t have permission to run Docker or the system has some outdated version of docker. To fix this, use the https://github.com/pigovsky/useful-scripts/blob/main/docker/install-docker.sh . It uninstalls any old versions of docker, and docker compose, installs their newest versions, and adds your user to the docker group.

2. Do not run any installation or deployment scripts with sudo or as the root user. Running scripts like deploy.sh or install.sh with elevated privileges can cause permission-related issues with configuration files not being read or updated by your usual user.

Run

./api/bin/install.sh

This script creates a so-called "dotfolder" configuration directory named .seqam_fh_dortmund_project_emulate in your home directory /home/<your_username>:

This folder contains an ssh key pair and important configuration files that need to be modified to suit your specific setup.

Please carefully read the explanation of Configuration files here before continuing. Also, you should read the page of the using setup to familiarize yourself with the configuration of these files given an example of a network before proceeding with the installation.

Modify Configuration Files

After running the api/bin/install.sh script, navigate to the dotfolder in your home directory:

cd ~/.seqam_fh_dortmund_project_emulate

• env: Modify the environment variables to match your specific network configuration.
• ScenarioConfig.json: Adjust this file to suit your deployment scenario and network topology.

Open the ~/.seqam_fh_dortmund_project_emulate/env file in your favorite text editor and set the SEQAM_CENTRAL_HOST variable to the address of the computer where you are going to install the Central components:

📝 *Note: This address should be the IP address or domain name of the machine (even one stated in the /etc/hosts folder). Do not use localhost as this will be unreachable by the containers.

Open the ~/.seqam_fh_dortmund_project_emulate/ScenarioConfig.json and remove, add or rename the sample devices with the real ones where you are going to install Distributed, Network components, vendor names and IP addresses of the routers. If you plan to have more distributed components in your setup (e.g. three servers), add each one of them individually to the file following the same format. Make sure you assign unique names.

Again, you can refer to the configuration files here, which are provided as an example of the network setup used in the demo.

Run

./bare-composes/generate-docker-composes.sh y

This step takes some time as it will build docker images for all the components. In the command, the last "y" option means that we would like to generate tar-balls ready to be deployed on computers without an internet connection.

The above script generates few subfolders in the bare-composes folder:

• seqam-central/
• seqam-distributed-event-manager/
• seqam-net-spy/
• seqam-network-event-manager/
• seqam-docker-compose-tree/

the content of these directories should be copied to the computers that shall run the Central, Distributed and Network components.

Go to the computer where you would like to deploy the central components and set up the Central Collector. It is recommended to follow the Signoz installation guide. Signoz can be installed in various ways, including Signoz Cloud; however, SeQaM uses the standalone version with Docker. You can run the central collector on a separate machine to improve performance.

In a directory of your choosing, clone the SigNoz repository and cd into it by entering the following commands:

git clone https://github.com/SigNoz/signoz.git

cd signoz

Apply the apply-me-on-new-signoz.diff patch there:

git apply <path-to-where-you-cloned-the-platform-repository>/emulate-edge-diagnostics-platform/apply-me-on-new-signoz.diff

This step is required to expose the clickhouse db endpoint to the platform.

🛠️ Configuration Recommendations:

• Update OpenTelemetry Collector Configuration: The OpenTelemetry collector operates in batches, meaning that information is collected but not exported until certain conditions are met. There are two parameters to configure: datasize and timeout. When either condition is fulfilled, the data is forwarded to the exporter. For faster data insertion into the database, it is crucial to configure these parameters in both the central and distributed collectors. For the central collector, locate the otel-collector-config.yaml file in signoz/deploy/docker and adjust it as follows:

processors:
  batch:
    send_batch_size: 100
    send_batch_max_size: 110
    timeout: 1ms

To run SigNoz, cd to the deploy/docker directory and enter the docker compose up command with:

• -d to run containers in the background:

cd deploy/docker

docker compose up -d

The central collector can be still separated as it is a completely isolated unit. This means it can run on a different machine than the core components. However, to simplify the setup it is better to put them together.

Copy the content of the bare-composes/seqam-central/ folder from the Builder computer (the machine where you have built the docker images) to the computer where you are going to deploy the Central components:

Go to the Central computer, navigate to the seqam-central folder and run

./load-image.sh seqam.central.tar.gz

it should load docker images of the Central components:

You can see the loaded docker images:

Once this step is completed, proceed to deploy the platform components:

docker compose up -d

It should start all the Central components:

⚠️ Warning:
The API component requires Signoz (clickhouse) to be running as it implements database migrations when starting. In case you have Signoz + Clickhouse in a different machine, make sure you change the environmental variables of the docker-compose.yaml file to match the IP address. as default the system is built supposing all central components are in the same machine.

Now navigate your web-browser to the IP-of-the-Central-computer:8000. A "chat" window should open. Type "hello" in the input field and press <ENTER>. You should see a response like below:

To stop the platform

docker componse down

⚠️ Warning:

After deploying the platform, any changes or updates to the configuration must be made in the seqam-central/config configuration folder on the computer running the Central components. Also, if you change, for example, the port of a distributed component, then you will have to manually adjust the docker compose in that distributed component to expose the same port. Please refer to the Managing Configuration Files section for detailed instructions on how to modify the settings.

Deploy your instrumented application client/server together with the distributed components. To get started, you can also test out the demo application. Check out the demo installation guide here.

Copy the content of the bare-composes/seqam-distributed-event-manager/ folder from the Builder computer to each device (UE/server) where you are going to deploy the Distributed component. Use scp to securely transfer these files:

scp -r bare-composes/seqam-distributed-event-manager/ user_name@distributed_device_IP:optional_directory

Make sure to replace user_name and distributed_device_IP with the appropriate username and IP address of a distributed machine:

Once the files are in place, go to the Distributed device, navigate to the seqam-distributed-event-manager directory and run

./load-image.sh seqam-distributed-event-manager.tar.gz

it should load the docker image:

Deploy the distributed components by running the following commands:

# From the seqam-distributed-event-manager directory
docker compose up -d

To stop the distributed component just run docker compose down on the same folder.

Enable GPU Metrics

⚠️ Warning: Applicable For NVIDIA GPUs Only.

1. Install NVIDIA Container Toolkit following the official guide.

2. Modify docker-compose.yml by uncommenting the following lines:

   deploy:
     resources:
       reservations:
         devices:
           - driver: nvidia
             count: all
             capabilities: [gpu]

Copy the content of the bare-composes/seqam-network-event-manager/ directory from the Builder computer to the machine/VM that will represent the Load Client:

Login on that computer, navigate to the seqam-network-event-manager directory, and load the docker image with

./load-image.sh seqam-network-event-manager.tar.gz

Deploy the Network components on by running the following command

# From the seqam-network-event-manager directory
docker compose up -d

To stop the component just run docker compose down on the same folder.

To simulate traffic and handle performance measurements, start one more instance of the Network component on some computer and add its IP address to the seqam-central/config/ScenarioConfig.json file under the server section on the Central computer like the following:

{
  "name": "load-server",
  "host": "load_server_IP"
}

This load-server will run iperf3 in the background, listening for incoming client connections for bandwidth testing.

The seqam-central/config/ExperimentConfig.json file on the Central computer contains a sample experiment. Please open it and update the experiment_name.

Run the experiment by typing in the SeQaM "chat" window in your browser:

start_module module:experiment_dispatcher

Now, using /experiments API on the Central component, you can see the experiment in the list of conducted experiments:

Using /experiments/{exp_name}/apps/{app_name} API we can get detailed information about spans that happened within every experiment segment:

📝 *Note: {exp_name} is the name you have just changed in the seqam-central/config/ExperimentConfig.json file {app_name} is the service name of your instrumented application with OpenTelemety.

After installing the SeQaM components, a hidden folder named .seqam_fh_dortmund_project_emulate is created in the user's home directory /home/<user_name>. This folder contains important configuration files, including environment variables, that can be modified to suit your specific setup. This configuration is taken to build the images you distribute afterward among the central, distributed, and network components.

After you have successfully installed it, navigate to the hidden configuration folder:

cd ~/.seqam_fh_dortmund_project_emulate

Here, you'll find various configuration files, such as environment variables and JSON configurations, which control various aspects of SeQaM.

To change or update any configuration (such as environment variables). Open the relevant configuration file (e.g., env) using a text editor of your choice:

nano env

Make the necessary changes in the file. Save the file and exit the editor. Once you have updated the configuration files, you need to build SeQaM components to apply the changes. For the first time you run the manager, the images will be built based on the configuration you have set. If you change them after, you can rather rebuild and replace the images or manually update the specific parts in the central, distributed, and network components.

Experiments consist of a set of commands triggered at a certain point in time.

📝 Note: More on commands here

⚠️ Warning: Currently, it is not possible to trigger two network events on the same devices one after the other due to a recovery limitation in iperf. The experiment dispatcher creates a thread per event request so they do not block the queue. However, it is recommended to add a small delay between concurrent events

Example:

{
  "eventList": [
   {
      "command": "cpu_load src_device_type:server src_device_name:svr101 cores:1 mode:inc load:80 time:4s",
      "executionTime": 100
    },
    {
      "command": "network_load src_device_type:router src_device_name:router1 interface:eth0 load:80 time:5s",
     "executionTime": 105
    }
  ]
}

Before running an experiment, there are a set of conditions you need to make sure your set-up satisfies. These are configured in the ExperimentConfig.json. The configuration is pretty simple and must be sequential.

The following file shows the syntax

{
  "experiment_name": "experiment_name",
  "eventList": [
    {
      "command": "command to run",
      "executionTime": time(in ms)
    },
    ...
    {
      "command": "exit",
      "executionTime": time(in ms)
    }
  ]
}

The experiment_name parameter is used to generate a file that contains the set of events performed in the experiment. Each event or combination of events is created as spans and thus stored in the database. The information about an experiment is accessible via the API. The data includes the command and the UTC time in UNIX format (microseconds) at which it was executed.

experiment_name and timestamps are the only ways we provide now to get the data collected during an experiment. Remember to change the experiment name each time you run a new experiment. If you forget to do so, you could still retrieve the data based on timestamps, but this will imply creating your query in the database.

The exit command exits the experiment dispatcher once all events have been dispatched. Make sure you add it at the end of the experiment to ensure that the trace completes and, thus, the data is accessible using the API.

{
  "experiment_name": "test_case_1",
  "eventList": [
    {
      "command": "cpu_load src_device_type:ue src_device_name:ue001 cores:5 load:20 time:60s",
      "executionTime": 8000
    },
    {
      "command": "memory_load src_device_type:ue src_device_name:ue001 workers:5 load:20 time:10s",
      "executionTime": 10500
    },
    {
      "command": "exit",
      "executionTime": 21000
    }
  ]
}

The executionTime is set in milliseconds and starts to count from the time the trigger signal is sent to the experiment dispatcher module. Be aware that if you want to trigger two events one after the other finishes (for example CPU load of 80% starting at t0 and finishing at t1, followed by a network load of 80% starting at t1), you have to consider adding up the duration of the first event to the executionTime of the second.

Example:

{
  "eventList": [
   {
      "command": "cpu_load src_device_type:server src_device_name:svr101 cores:1 mode:inc load:80 time:40s",
      "executionTime": 100
    },
    {
      "command": "network_load src_device_type:router src_device_name:router1 interface:eth0 load:80 time:50s",
      "executionTime": 40100
    }
  ]
}

Here,

• t0=100
• duration of the first event 40s (40000ms). Therefore, the time when the event will finish is t1 = 40100

For generating concurrent execution (two or more events running together during a period of time), multiple commands shall be triggered before the time of the prior event elapses. Each command runs independently, meaning the time frame for each command can overlap based on their respective durations

Example:

{
  "eventList": [
   {
      "command": "cpu_load src_device_type:server src_device_name:svr101 cores:1 mode:inc load:80 time:80s",
      "executionTime": 10000
    },
    {
      "command": "network_load src_device_type:router src_device_name:router1 interface:eth0 load:80 time:60s",
      "executionTime": 20000
    }
  ]
}

Here,

• The first event, cpu_load, starts at t0 = 10000 ms and runs for 80 seconds (80000 ms).
• The second event, network_load, starts at the same time (t1 = 20000 ms) and runs for 60 seconds (60000 ms).

As both events are running concurrently, their execution will overlap over a time frame. While getting data from the API, you will receive the spans during this overlapping in a separate portion of the response.

📝 Note: Here are some points you need to consider before running an experiment:

1. In the laboratory deployment scenario, all connections should be made and tested, and IP addresses should be updated in the ScenarioConfig.json file.
2. You have properly configured the .env file.
3. The edge application must share the same machine with the distributed components
4. The distributed components are up and running (Refer installation guide)
5. The network traffic generators have been attached to your scenario and are up and running (Refer installation guide)
6. The device types and names are properly set in the ScenarioConfig.json file and in the commands described in the ExperimentConfig.json
7. You have created/modified the ExperimentConfig.json accordingly.

The experiment dispatcher waits for a trigger event from the web console to start.

start_module module:experiment_dispatcher

📝 Note: When you run multiple experiments, make sure you change the name of each of them.

⚠️ Danger: Modifying the ModuleConfig.json file is not recommended unless you specifically need to change the port or other deployment settings of the distributed source code (e.g., change or add endpoints in the core components)

The Module Configuration file must have the following format:

{
  "modules": [
    {
      "*module_name*": {
        "name": "module_display_name",
        "description": "module_description",
        "port": "module_port",
        "host": "module_host",
        "paths": [
          {
            "*action_name*": {
              "endpoint": "/path_endpoint/"
            }
          },
          ...
        ]
      }
    },
    ...
  ]
}

The core components of SeQaM can be customized by modifying the ModuleConfig.json file. IP addresses and ports can be configured in this file when using the agent deployment method (you directly run each component from a terminal, with the prior installation of all its dependencies [not recommended]) or by modifying the .env file when using the containerized version.

Each module is represented as a JSON with the following parameters:

• name: It is the name of the module. This is a hardcoded parameter, therefore, it is not worth changing it, as other modules use this name to generate the endpoint to which send their request from this file
• description: A small and simple description of each module
• port: The port the module listens to when it is deployed. This can be configured according to your needs. It is obtained from the environment variables. The system will automatically retrieve the port from here, so there is no problem if you prefer to change it.
• host: In case you prefer to distribute the SeQaM central components across different physical devices, you can add their IP address here, so all the other modules can communicate with it when required. If all are in the same host, then configure it as 127.0.0.1.

⚠️ Warning: Do not use localhost as DNS resolution takes unnecessary time. When configuring the .env file, the host IPs and ports are obtained from the Environment Variables.

• paths: The path is characterized by an action and an endpoint. Actions are hardcoded elements that are used by the other components of the manager to make a call to a certain action within the component. These values should not be changed, but you are free to add new ones if you need them. The endpoint can be configured according to your criteria, as the components read this value before making the REST request. However, if not required, just leave them as they are set.

{
    "modules":[
       {
          "console":{
             "name":"Console",
             "description":"Console module to input commands",
             "port":0,
             "host":"0.0.0.0",
             "paths":[
                
             ]
          }
       },
       {
          "command_translator":{
             "name":"Command Translator",
             "description":"Get raw commands and forward them to orchestrators in json format",
             "port": 8001,
             "host": "172.22.174.157",
             "paths":[
                {
                   "translate":{
                      "endpoint":"/translate/"
                   }
                }
             ]
          }
       },
       {
          "event_orchestrator":{
             "name":"Event Orchestrator",
             "description":"Get event requests",
             "port": 8002,
             "host": "172.22.174.157",
             "paths":[
                {
                   "event":{
                      "endpoint":"/event/"
                   }
                }
             ]
          }
       },
      {
        "experiment_dispatcher":{
           "name":"Experiment Dispatcher",
           "description":"Executes the configured experiment",
           "port": 8004,
           "host": "172.22.174.157",
           "paths":[
              {
                 "start":{
                    "endpoint":"/experiment/init/"
                 }
              }
           ]
        }
     }
    ]
 }

The Scenario Configuration file must have the following format:

{
  "distributed": {
    "*device_type*": [
      {
        "name": "*device_name*",
        "description": "*device description*",
        "port": *agent-port*,
        "host": "*host*",
        "ssh_port": *ssh-daemon-port*,
        "ssh_user": "*ssh-username*",
        "paths": [
          {
            "*action*": {
              "endpoint": "/*path*/"
            },
            ...
          }
        ]
      },
      ...
    ]
  }
}

where device_type is the name of device category; it should represent either src_device_type or dst_device_type; possible device types are ue (user equipment), server, and router;

device_name is the computer-friendly identifier of the device, something like the hostname; for example device_names are ue001, svr101, svr102, ntw_agent; the device name should be unique within every device_type set. This name is used when triggering the commands.

agent-port is an optional integer port where either Distributed Event Manager Agent or Network Event Manager Agent can be listening on; it is optional because the host can be controlled using either agent or an agentless ssh approach. This is required if you plan to run the ssh command;

the host can be specified by IPv4 literal, or IPv6 literal.

ssh-daemon-port is an optional integer port where an ssh daemon can be listening.

This file configures your scenario for deploying SeQaM.

{
    "distributed":
    {
        "ue":[
            {
                "name":"ue001",
                "description":"User Equipment 1",
                "port": 9001,
                "host": "192.168.1.16",
                "ssh_port": 22,
                "ssh_user": "ubuntu",
                "paths":[
                   {
                      "event":{
                         "endpoint":"/event/"
                      },
                      "cpu_load":{
                        "endpoint":"/event/stress/cpu_load"
                     },
                     "memory_load":{
                       "endpoint":"/event/stress/memory_load"
                    }
                   }
                ]
            }
        ] ,
        "server":[
            {
                "name":"svr001",
                "description":"Network Load Server",
                "port": 9001,
                "host": "172.22.174.175",
                "paths":[
                   {
                      "event":{
                         "endpoint":"/event/"
                      },
                      "cpu_load":{
                        "endpoint":"/event/stress/cpu_load"
                     },
                     "memory_load":{
                       "endpoint":"/event/stress/memory_load"
                    }
                   }
                ]
            }
        ],
        "router":[
         {
             "name":"ntw_agent",
             "description":"Endpoint of the agent that runs in the same network emulator VM. The name is hardcoded, and this section is to be deprecated",
             "port": 8887,
             "host": "172.22.174.175",
             "paths":[
                {
                   "network_bandwidth":{
                      "endpoint":"/event/network/bandwidth"
                   },
                   "network_load":{
                     "endpoint":"/event/network/load"
                  }
                }
             ]
         }
      ]
    }
}

For a detailed explanation of how to configure the ScenarioConfig.json file for network topology, check out the demo setup guide.

📝 Note:

• The ntw_agent and LoadClient are the same component if you still use the deprecated commands for network load. This agent runs the network event manager and is responsible for managing network-related events. In the new version of the command, where you define source and destination, this is not needed

The following table explains the environment variables used in the SeQaM configuration:

Depending on your setup, you may need to configure one or more Distributed Event Managers and Network Event Managers. In the following example, there are two Distributed Event Managers and one Network Event Manager

SEQAM_CENTRAL_HOST=172.22.229.149
DATABASE_ENDPOINT="$SEQAM_CENTRAL_HOST"
OTLP_URL="$DATABASE_ENDPOINT":4317
API_HOST="$SEQAM_CENTRAL_HOST"
API_PORT=8000
COMMAND_TRANSLATOR_HOST="$SEQAM_CENTRAL_HOST"
COMMAND_TRANSLATOR_PORT=8001
EVENT_ORCHESTRATOR_HOST="$SEQAM_CENTRAL_HOST"
EVENT_ORCHESTRATOR_PORT=8002
EXPERIMENT_DISPATCHER_HOST="$SEQAM_CENTRAL_HOST"
EXPERIMENT_DISPATCHER_PORT=8004