Only for ACX7100, In MPLS networks when Containers LSP are used, the Auto-bandwidth feature does not display accurate counters if Telemetry Feature is enabled with following subscription path:"/junos/services/label-switched-path/usage/".
When Telemetry is enabled on the router and customer tries to subscribe to unsupported telemetry sensors, the subscription happens but PFE is still not correctly programmed to understand the telemetry request so it potentially cause incorrect data read in the LSP counters causing the incorrect LSP auto-bandwith adjustment.
root@Ubuntu:/etc/telegraf# systemctl status telegraf
● telegraf.service - Telegraf
Loaded: loaded (/lib/systemd/system/telegraf.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-06-06 12:40:15 UTC; 6s ago
Docs: https://github.com/influxdata/telegraf
Main PID: 155106 (telegraf)
Tasks: 6 (limit: 2257)
Memory: 19.1M
CGroup: /system.slice/telegraf.service
└─155106 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
Jun 06 12:40:16 Ubuntu telegraf[155106]: 2024-06-06T12:40:16Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
Jun 06 12:40:17 Ubuntu telegraf[155106]: 2024-06-06T12:40:17Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
Jun 06 12:40:18 Ubuntu telegraf[155106]: 2024-06-06T12:40:18Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
Jun 06 12:40:19 Ubuntu telegraf[155106]: 2024-06-06T12:40:19Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
Jun 06 12:40:20 Ubuntu telegraf[155106]: 2024-06-06T12:40:20Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
Jun 06 12:40:21 Ubuntu telegraf[155106]: 2024-06-06T12:40:21Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
You can verify the issue by checking the PPS rate in the AE interface facing the egress side of the LSP:
user@Router> show interfaces ae1 | match rate
Input rate : 95417007456 bps (14908922 pps)
Output rate : 95417007456 bps (14908920 pps)
From following command you can see that the Bandwidth Adjustment is incorrect set:
user@Router> show mpls lsp ingress name TEST-LSP-1 detail | match Band
Autobandwidth
Max AvgBW util: 16.1915Gbps, Bandwidth Adjustment in 594 second(s).
Bandwidth: 19.4308Gbps. >>> Incorrect value, it should display 95Gbps or similar value reported on the AE interface.
The H/W counters allocated for LSP stats are getting cleared on read for about every 5 to 10 seconds as telemetry is querying for LSP stats, hence when polling LSP Stats happens every 60 seconds from the control plane, PFE is reporting less(incorrect) statistics which is causing incorrect LSP Bandwidth adjustment.
Telemetry feature for ACX7100 is still under review and will be fully operational till 25.1 release. At this point, we recommend our customer to avoid setting Telemetry configuration or subscribe to any path since it will cause random issues in the PFE.
As per date June 29, our DEV team is working on the defensive fix for this issue via PR: 1810260
Fix:
Sensors subscription still happens, but nothing will be displayed. We need to unset the "clear on read" flag while reading H/W LSP stats counters when telemetry is querying for LSP stats.
Our developer team is working on this issue via PR1810260, meaning that the subscription will occur but sensor will not work.
This is since the telemetry feature for ACX7100 is still under review and will be fully operational till 25.1 release.
At this point after opting for fix 1 that auto-bandwidth will not be affected even if customers mistakenly enable the unsupported sensors.
Workarround: Deactivate that particular sensor, so that sensor subscription will not happen. Customer can use this as workarround while the fix gets added in 23.2R2-S2.
We recommend our customer to avoid setting Telemetry configuration or avoid the path "/junos/services/label-switched-path/usage/" subscription.