telemetry_under_churn_test - openconfig/featureprofiles GitHub Wiki
Verify that gNMI state streaming (specifically interface counters and AFT states) remains stable and within latency SLOs while the device is actively processing high-rate routing/gRIBI updates. This test ensures that the gNMI session does not disconnect or report sync loss during control plane churn, 100% of telemetry updates are received within the configured interval without starvation, and interface counters match the actual traffic volume sent by the ATE.
- Configure IPv4 and IPv6 addresses on DUT and ATE on all 4 links:
- Link 1 (
port1): ATE 192.0.2.1/30 & 2001:db8:1::1/126, DUT 192.0.2.2/30 & 2001:db8:1::2/126 - Link 2 (
port2): ATE 192.0.2.5/30 & 2001:db8:2::1/126, DUT 192.0.2.6/30 & 2001:db8:2::2/126 - Link 3 (
port3): ATE 192.0.2.9/30 & 2001:db8:3::1/126, DUT 192.0.2.10/30 & 2001:db8:3::2/126 - Link 4 (
port4): ATE 192.0.2.13/30 & 2001:db8:4::1/126, DUT 192.0.2.14/30 & 2001:db8:4::2/126
- Link 1 (
- Configure BGP on the ATE to generate traffic destined to the routes that will be churned via gRIBI. DUT AS
65001, ATE AS65002. - Establish a gRIBI session between ATE (acting as gRIBI controller) and DUT using
SINGLE_PRIMARYredundancy mode, andPRESERVEpersistence.
- Step 1 - Generate DUT configuration
- Configure interfaces on the DUT with IPv4 and IPv6 addresses for Link 1 (
port1) and Link 2 (port2).
- Configure interfaces on the DUT with IPv4 and IPv6 addresses for Link 1 (
- Step 2 - Push configuration to DUT using gNMI Set with REPLACE option.
- Step 3 - Event-Driven Validation:
- Wait for
/interfaces/interface[name=port1]/state/oper-statusand/interfaces/interface[name=port2]/state/oper-statusto equalUPvia gNMI Subscribe. - Wait for
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-stateto equalESTABLISHEDvia gNMI Subscribe.
- Wait for
- Step 4 - Establish gNMI subscriptions on the DUT for interface counters (
/interfaces/interface/state/counters) and AFT states (/network-instances/network-instance/afts) with a 10-second update interval (SAMPLE). - Step 5 - Inject a baseline batch of 100 IPv4 and 100 IPv6 routes via the gRIBI session on Link 2.
- Step 6 - Wait for the presence of specific prefixes in
/network-instances/network-instance/afts/ipv4-unicast/ipv4-entry/state/prefixto guarantee routes are fully programmed. - Step 7 - Start ATE traffic from Link 1 (
port1) to Link 2 (port2) matching the programmed gRIBI routes. - Step 8 - Continuous gRIBI churn: Inject a continuous stream of gRIBI route modifications (add/delete/update IPv4 and IPv6 NextHopGroups and Entries) for the baseline routes from the ATE gRIBI controller.
-
Guidance on gRIBI Churn: To achieve "churn" without disrupting the traffic flow entirely, you can perform gRIBI
ModifyRPCs in a continuous loop:-
NHG Updates: Modify an existing
NextHopGroup(NHG) by adding or removing a secondaryNextHop, or by changing the weight of aNextHop. -
Entry Swaps: Change an
IPv4EntryorIPv6Entryto point to a different, pre-programmedNextHopGroupthat still routes valid traffic to the ATE. -
Add/Delete Cycles: Rapidly send
DELETEand thenADDoperations for a subset of the programmed prefixes.
-
NHG Updates: Modify an existing
-
Guidance on gRIBI Churn: To achieve "churn" without disrupting the traffic flow entirely, you can perform gRIBI
- Step 9 - Validation with pass/fail criteria:
-
Pass Criteria:
- Subscription Stability: The gNMI session does not disconnect or report sync loss during control plane churn.
- On-time Delivery: Verify the
timestampin the gNMISubscribeResponseto ensure telemetry updates arrive exactly at the 10-second interval (no starvation). - Data Accuracy: The delta of
in-unicast-pktsandout-unicast-pktsreported by gNMI matches the ATE Tx/Rx rate within a 1% tolerance. - Lossless Validation: Total ATE Tx packets == ATE Rx packets (0% packet drop).
-
Pass Criteria:
- Step 1 - Generate DUT configuration
- Configure interfaces on the DUT with IPv4 and IPv6 addresses for all 4 links (
port1throughport4).
- Configure interfaces on the DUT with IPv4 and IPv6 addresses for all 4 links (
- Step 2 - Push configuration to DUT using gNMI Set with REPLACE option.
- Step 3 - Event-Driven Validation:
- Wait for
oper-statusto equalUPon all 4 interfaces via gNMI Subscribe. - Wait for BGP
session-stateto equalESTABLISHEDon all 4 links via gNMI Subscribe.
- Wait for
- Step 4 - Establish gNMI subscriptions on the DUT for interface counters and AFT states on all links with a 10-second update interval (
SAMPLE). - Step 5 - Scale Up Route Injection: Program 10,000+ IPv4 and 10,000+ IPv6 routes with ECMP next-hops spread across
port2,port3, andport4via gRIBI. - Step 6 - Wait for the presence of the scaled prefixes in
/network-instances/network-instance/afts/ipv4-unicast/ipv4-entry/state/prefixand/network-instances/network-instance/afts/ipv6-unicast/ipv6-entry/state/prefixto guarantee full programming. - Step 7 - Start ATE traffic from
port1load-balanced across the scaled gRIBI routes toport2,port3, andport4. - Step 8 - Scale Up gRIBI Churn: Continuously modify and delete routes at a high rate (e.g., 1,000 operations/sec) from the ATE gRIBI controller.
-
Guidance on gRIBI Churn: In this scaled scenario, leverage the ECMP paths across
port2,port3, andport4. Send batches of gRIBIModifyrequests that continuously shift theNextHopGroupassignments for thousands of prefixes, or rapidly add/removeNextHopmembers within the NHGs to force massive AFT updates across the DUT's forwarding pipeline.
-
Guidance on gRIBI Churn: In this scaled scenario, leverage the ECMP paths across
- Step 9 - Validation with pass/fail criteria:
-
Pass Criteria:
- Same as 1.30.1: No gNMI disconnects, strict 10-second update cadence per timestamp validation, counter deltas match ATE rates within 1%, and zero traffic loss during ECMP churn.
-
Pass Criteria:
- Step 1 - While the scaled high-rate traffic and gRIBI churn from
gNMI-1.30.2are actively running, simulate a sudden gRIBI controller crash or network disconnect from the ATE side. - Step 2 - Observe the active gNMI telemetry subscriptions for interface counters.
- Step 3 - Validation with pass/fail criteria:
-
Pass Criteria:
- Telemetry Isolation: The gNMI telemetry session remains completely unaffected by the gRIBI session drop.
- No Sync Loss: The gNMI session does not drop, reconnect, or report sync loss.
- Continuous Accuracy: Interface counters (
in-unicast-pkts,out-unicast-pkts,in-octets,out-octets) continue to stream at the accurate configured interval without interruption. - Route Persistence: Since gRIBI was configured with
PRESERVEpersistence, AFT states and traffic forwarding should remain unaffected during the disconnect.
-
Pass Criteria:
{
"openconfig-interfaces:interfaces": {
"interface": [
{
"name": "Ethernet1",
"config": {
"type": "iana-if-type:ethernetCsmacd",
"description": "DUT to ATE link 1",
"enabled": true
}
}
]
}
}paths:
/interfaces/interface/state/counters/in-unicast-pkts:
/interfaces/interface/state/counters/out-unicast-pkts:
/interfaces/interface/state/counters/in-octets:
/interfaces/interface/state/counters/out-octets:
/interfaces/interface/state/oper-status:
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state/session-state:
/network-instances/network-instance/afts/next-hop-groups/next-hop-group/state/id:
/network-instances/network-instance/afts/next-hops/next-hop/state/index:
/network-instances/network-instance/afts/ipv4-unicast/ipv4-entry/state/prefix:
/network-instances/network-instance/afts/ipv6-unicast/ipv6-entry/state/prefix:
rpcs:
gnmi:
gNMI.Set:
union_replace: true
gNMI.Subscribe:
on_change: true
sample: true- FFF