PFC Watchdog Design - mykolaf/SONiC GitHub Wiki
| Rev | Date | Author | Change Description |
|---|---|---|---|
| 0.1 | Marian Pritsak | Initial version |
This document provides general information about the PFC Watchdog feature implementation in SONiC. PFC watchdog is designed to detect and mitigate PFC storm received for each port. PFC pause frames are used in lossless Ethernet to pause the link partner from sending packets. Such back-pressure mechanism could propagate to the whole network and cause the network stop forwarding traffic. PFC watchdog is to detect abnormal back-pressure caused by receiving excessive PFC pause frames, and mitigate such situation by disabling PFC caused pause temporarily. PFC watchdog has three functional blocks, i.e. detection, mitigation and restoration.
This document describes the high level design of the PFC WD feature.
| Definitions/Abbreviation | Description |
|---|---|
| PFC | Priority Flow Control |
| WD | Watchdog |
| ACL | Access Control List |
- Support PAUSE duration counters (M)
- Support egress ACL for drop mitigation action (S)
; Defines PFC WD configuration on a port
key = PFC_WD_TABLE:ifname ; configuration for watchdog on port
; field = value
state = "enabled"/"disabled" ; state of WD as configured by user
t0 = 1*3DIGIT ; t0 pause duration in msecs time after which
; queue is considered to be in PFC
; storm, and watchdog action is triggered.
t1 = 1*3DIGIT ; t1 time in msecs after which
; queue in PFC storm state is
; restored if no pause frames were received.
action = "drop"/"forward" ; action taken by watchdog in case
; of PFC storm detection.
Orchagent can have different strategies or criteria for storm detection, hence it needs to tell syncd which counters to poll.
; Defines schema for queues that must be polled by syncd, and corresponding counters
key = "PFC_WD_STATE":""queueId" ; WD queue entry
; field = value
PORT_COUNTER_ID_LIST = 1*64VCHAR ; list of ',' separated counter IDs
QUEUE_COUNTER_ID_LIST = 1*64VCHAR ; list of ',' separated counter IDs
; Defines schema for queue counters that are updated by PFC WD
key = "COUNTERS":""queueId" ; WD queue entry
; field = value
PFC_WD_QUEUE_STATS_STORM_DETECTED = 1*4DIGIT ; deadlock counter
PFC_WD_QUEUE_STATS_STORM_RESTORED = 1*4DIGIT ; restoration counter
PFC_WD_QUEUE_STATS_TX_PACKETS = 1*20DIGIT ; total packets transmitted during storm
PFC_WD_QUEUE_STATS_TX_DROPPED_PACKETS = 1*20DIGIT ; total Tx packets dropped due to storm
PFC_WD_QUEUE_STATS_RX_PACKETS = 1*20DIGIT ; total packets received during storm
PFC_WD_QUEUE_STATS_RX_DROPPED_PACKETS_LAST = 1*20DIGIT ; total Rx packets dropped due to storm
PFC_WD_QUEUE_STATS_TX_PACKETS_LAST = 1*20DIGIT ; packets transmitted during last storm
PFC_WD_QUEUE_STATS_TX_DROPPED_PACKETS_LAST = 1*20DIGIT ; Tx packets dropped due to last storm
PFC_WD_QUEUE_STATS_RX_PACKETS_LAST = 1*20DIGIT ; packets received during last storm
PFC_WD_QUEUE_STATS_RX_DROPPED_PACKETS_LAST = 1*20DIGIT ; Rx packets dropped due to last storm
As different vendors support different counters, there must be a way to let every ASIC vendor decide how to tell if queue is stormed. For instance, a possible criteria could be one based on pause duration counter:
(SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES > 0 && SAI_QUEUE_STAT_PACKETS.current - SAI_QUEUE_STAT_PACKETS.last == 0 && SAI_PORT_STAT_PFC_[queue]_RX_PKT.current - SAI_PORT_STAT_PFC_[queue]_RX_PKT > 0)
||
(SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES == 0 && SAI_QUEUE_STAT_PACKETS.current - SAI_QUEUE_STAT_PACKETS.last == 0 && SAI_PORT_STAT_PFC_[queue]_RX_PAUSE_DURATION.current == SAI_PORT_STAT_PFC_[queue]_RX_PAUSE_DURATION.last + t0 * delta)
// delta is a percentage of time that queue had to be paused (e. g. 0.9)Those criteria are coded in .lua script for Redis DB and called periodically based on configured timers by orchagent.
As different platforms might choose different actions to be performed for mitigation and restoration, PfcWdAction class is defined to provide common interface for these handlers:
class PfcWdAction
{
virtual sai_status_t MitigateHandler(sai_object_t port, uint32_t queueId);
virtual sai_status_t RestoreHandler(sai_object_t port, uint32_t queueId);
};Currently drop and forward actions are defined and provide corresponding mark and unmark handlers.
Watchdog settings are specified per port, in the device minigraph under DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface.
All configuration goes under DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog tag. if its absent, watchdog is considered to be disabled for that port.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/Action defines mitigation action to be taken in case of pause storm.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/DetectionTime defines time interval for storm detection.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/RestorationTime defines time interval for restoration queue from storm.
...
<DeviceInfos>
...
<DeviceInfo>
<EthernetInterfaces>
...
<EthernetInterface>
<InterfaceName>Ethernet1</InterfaceName>
...
<PfcWatchdog>
<Action>drop</Action>
<DetectionTime>200</DetectionTime>
<RestorationTime>5000</RestorationTime>
</PfcWatchdog>
...
</EthernetInterface>
...
</EthernetInterfaces>
</DeviceInfo>
</DeviceInfos>
There is a set of external events that can fully or partially invalidate current state of watchdog.
In case of resetting counters PFC WD is in undefined state and should skip one polling interval. SONiC does not use API to reset counters on ASIC, so this event is ignored.
In case if port's state changes to DOWN, all queues marked as stormed, are restored.
Queue configuration must be going through PFC WD proxy in order to make it update its internal state. If PFC is disabled on a queue that was marked as stormed, queue will be restored.
In order to provide user an ability to set/view PFC WD configuration/statistics, pfcwatchdog CLI tool should provide following functionality:
- Show watchdog configuration (per port).
- Show watchdog statistics (per port/queue).
- Enable watchdog on a specified port(s).
- Disable watchdog on a specified port(s).
CLI tool should provide the ability to filter any option by a list of ports on which command is being applied, or apply to all ports if the list is not provided.
All port counters are stored in Counters DB. In order to have all required information from hardware, watchdog needs to extend it by making syncd read values from SAI: it subscribes queues for polling in WD database. syncd will call apropriate .lua script upon every counters update to check if queues changed their state and notify orchagent upon any change.
Orchagent is subscribed to syncd notifications of queue deadlock, and is supposed to apply configured action (drop/forward) upon receiving notification that queue became locked, or restore queue when opposite notification type is received.
Following mitigation handler disables PFC on marked queue and sets its reserved buffer to 0.
Following restoration handler returns reserved buffer value to initial value and enables PFC on unmarked queue.
Following mitigation handler disables PFC on stormed queue. It will no more respect pause frames from link partner, and forward all packets.
Following restore handler will reenable PFC on a queue so that it will continue to work in lossless mode as configured by user.
