PFC Watchdog Test Plan - mykolaf/SONiC GitHub Wiki

Overview

The purpose is to test the functionality of PFC Watchdog on the SONIC switch DUT on Tor and Spine ports, closely resembling production environment. The test assumes all necessary configuration, PFC Watchdog, and BGP routes, are already pre-configured on the SONIC switch before test runs.

Scope

The test is targeting a running SONiC system with fully functioning configuration. Purpose of the test is to verify a SONiC switch system correctly detecting PFC storm, reacting on it according to the configuration.

Related DUT CLI commands

Manual PFC WD configuration can be done using swssconfig utility in swss container.

swssconfig <json-file to apply>

Test structure

Setup configuration

PFC WD configuration should be created on the DUT before running the test. The test itself can be run using SONiC test.yml playbook with tag pfc_wd.

Scripts for generating configuration on SONIC

For the cases which test configuration, there will be a wfc_wd.json.j2 template.

Ansible scripts to setup and run test

pwd_wd.yml

pfc_wd.yaml will do the following:

  1. Generate JSON files and apply them on the switch.
  2. Run test.
  3. Clean up dynamic configuration and temporary configuration on the DUT.

PFC WD test consists of a number of subtests, and each of them will include the following steps:

  1. Run lognanalyzer 'init' phase
  2. Run PFC WD Sub Test
  3. Run loganalyzer 'analyze' phase

Subtests will test either a configuration or watchdog functionality.

Setup of DUT switch

Setup of SONIC DUT will be done by Ansible script. During setup, Ansible will copy JSON file containing configuration for Watchdog to the swss container on the DUT. the swssconfig utility will be used to push the configuration to the SONiC DB. Data will be consumed by orchagent.

JSON Sample:

pfc_wd.json

[
    {
        "PFC_WD_TABLE:Ethernet0": {
            "action": "drop",
            "detection_time": "500",
            "restoration_time": "5000"
        },
        "OP": "SET"
    }
]

PFC Frames Generation

PFC storm might be generated in 2 ways:

  • Directly on a fanout switch;
  • Generated on a server, encapsulated into IP packet and decapsulated by a fanout;

In order to verify both actions (drop/forward), traffic will be sent through a stormed port and expected to be forwarded or dropped respectively.

##Test cases

Each test case will be additionally validated by the loganalizer utility.

Some cases will add PFC WD configuration at the beginning and remove them at the end. They are grouped into two categories: configuration tests and traffic tests.

Configuration Tests

Test case #1 - Configure forward action

Test objective

Verify that watchdog with forwarding action will be successfully configured and then removed from the specified port.

Test steps

  • Initialize log analyzer.
  • Apply JSON file with WD configuration with forwarding action.
  • Verify that log has the message saying that WD started on specified port.
  • Initialize log analyzer.
  • Apply JSON file with removal of WD configuration with forwarding action.
  • Verify that log has the message saying that WD stopped on specified port.

Test case #2 - Invalid action

Test objective

Verify orchagent correctly handles invalid WD action field.

  • Initialize log analyzer.
  • Apply JSON file with WD action other than "drop"/"forward".
  • Verify error in logs saying "Invalid PFC WD action".

Test case #3 - Invalid detection time

Test objective

Verify orchagent correctly handles invalid "detection_time" field.

  • Initialize log analyzer.
  • Apply JSON file with WD mitigation_time other than unsigned int.
  • Verify error in logs saying "Invalid PFC WD action".

Test case #4 - Invalid restoration time

Test objective

Verify orchagent correctly handles invalid "restoration_time" field.

  • Initialize log analyzer.
  • Apply JSON file with WD restoration_time other than unsigned int.
  • Verify error in logs saying "Invalid PFC WD action".

Functional Tests

Test case #5 - Trigger forward action

Test objective

Verify that WD is triggered under PFC storm and packets are forwarded from the interface.

  • Initialize log analyzer.
  • Apply JSON file with WD configuration with forwarding action.
  • Verify that log has the message saying that WD started on specified port.
  • Start sending traffic via the specified port.
  • Initiate PFC storm on the specified port.
  • Verify that traffic is forwarded.
  • Stop PFC storm.
  • Verify that logs contain the message that PFC WD was triggered on specified port/queue.
  • Verify that logs contain the message that queue was restored from the storm.
  • Remove PFC WD configuration from the port.

Test case #6 - Trigger drop action

Test objective

Verify that WD is triggered under PFC storm and packets are drop on the interface.

  • Initialize log analyzer.
  • Apply JSON file with WD configuration with the drop action.
  • Verify that log has the message saying that WD started on specified port.
  • Start sending traffic via the specified port.
  • Verify that traffic is forwarded.
  • Initiate PFC storm on the specified port.
  • Verify that traffic is dropped.
  • Stop PFC storm.
  • Verify that logs contain the message that PFC WD was triggered on specified port/queue.
  • Verify that logs contain the message that queue was restored from the storm.
  • Remove PFC WD configuration from the port.

Test case #7 - Timers

Test objective

Verify that WD detects storm and restores queue within specified time constraints.

  • Initialize log analyzer.
  • Apply JSON file with WD configuration.
  • Initiate PFC storm on the specified port.
  • Stop PFC storm.
  • Verify that WD was triggered within "mitigation_time".
  • Verify that queue was restored after "restoration_time".
  • Remove PFC WD configuration from the port.

Test case #8 - Resources

Test objective

Verify that WD is able to mitigate storm on all lossless queues of all ports

  • Initialize log analyzer.
  • Apply JSON file with WD configuration.
  • Simulate PFC storm on all queues and ports using DEBUG_STORM field in DB.
  • Verify that WD was triggered on all queues.
  • Stop PFC storm.
  • Remove PFC WD configuration from the port.

Test must be performed for both drop and forward actions.