Description

Changing port speed on ACX7100-32C will flap all interfaces in the same PHY group. 

 

Symptoms

Changing speed of any port of same Port Group. When port speed is changed on ACX7100-32C all interfaces of port group flaps.

Here is LAB setup:

 

ACX7100-A-et-0/0/11<>et-0/0/11-ACX7100-B
ACX7100-A-et-0/0/15<>et-0/0/15-ACX7100-B
ACX7100-A-et-0/0/19<>et-0/0/19-ACX7100-B

if you change the speed of interface et-0/0/8 from 100g to 40g or from 40g to 100g interface et-0/0/11 flaps.
if you change the speed of interface et-0/0/12 from 100g to 40g or from 40g to 100g interface et-0/0/15 flaps.
if you change the speed of interface et-0/0/16 from 100g to 40g or from 40g to 100g interface et-0/0/19 flaps.

Issue related with the port-group that is on the same PHY.
There is no config on ACX7100-32C on interfaces et-0//0/11 , 15 and 19.

There is no need to have a 40G transceiver inserted on ACX, just the fact of changing the sport speed trigger the issue.

For instance:

root@acx7100-A# run show interfaces et-0/0/11 | grep flapped
Last flapped : 2023-09-11 11:08:21 -03 (00:09:07 ago)

[edit]
root@acx7100-A# set interfaces et-0/0/8 speed 40g


[edit]
root@acx7100-A# show | compare
[edit interfaces]
+ et-0/0/8 {
+ speed 40g;
+ }

[edit]
root@acx7100-A# commit
commit complete

[edit]
root@acx7100-A# run show interfaces et-0/0/11 | grep flapped
Last flapped : 2023-09-11 11:17:52 -03 (00:00:01 ago)


There is no config on physical interfaces that are running on 100G.

root@acx7100-A show interfaces | display set
set interfaces et-0/0/8 speed 40g
set interfaces re0:mgmt-0 unit 0 family inet address 172.25.0.2/24


root@acx7100-A# run show interfaces terse | except down | grep et-
et-0/0/11 up up
et-0/0/11.16386 up up multiservice
et-0/0/15 up up
et-0/0/15.16386 up up multiservice
et-0/0/19 up up
et-0/0/19.16386 up up multiservice

On this blog you will find the information about the port groups.

https://community.juniper.net/blogs/pankaj-kumar/2022/08/16/acx7100-deepdive

For the QSFP ports from 0 to 31, 4 subsequent ports constitute a group.

PG 0 > Ports 0 - 3
PG 1 > Ports 4 - 7
PG 2 > Ports 8 - 11
PG 3 > Ports 12 - 15
PG 4 > Ports 16 - 19
PG 5 > Ports 20 - 23
PG 6 > Ports 24 - 27
PG 7 > Ports 28 - 31

 

 

Solution

This is a platform limitation.

 

In ACX7100-32C , it supports speed having 3 different PLL clocks
1x10,4x10,40G ,
4x25,1x25, 
2x50 and 1x100. 

As per BRCM to dynamic change in speed which allow change in PLL clock in a port group have to use APIs which makes other ports in a PG flap.

Similar case with ACX7100-48L

Here each PG supports speed having 3 PLL clocks 
1x10G
1x50G
1x25G
to supports dynamic change in speed and PLL clock have to use BRCM API which makes other ports in the PG flap.

This is a limitation in BRCM side. and same is mentioned in their document as well  - 
-----------
Static configuration of a 1  200GbE port (four lanes, VCO = 26.5625G), and 4  10GbE ports (four lanes, VCO = 20.6250G)
results in {PLL1 = 26.5625G, PLL0 = 20.6250G}.
A dynamic configuration that replaces the 200GbE port with a 100GbE port (four lanes, VCO = 25.78125G) results in an
error indication because PLL1 should be updated, causing a port-down event on the 10GbE ports as well.
If it is acceptable to allow the 10GbE ports to go down and back up, they should be added to the allocation update (by using
bcm_port_resource_multi_set).
----------- 

 

Modification History

2023-11-06 : Article Created

Related Information

2023-0911-767453

PR   : 1773068 [Confidential] - CHANGING SPEED OF ONE INTERFACE, TRIGGER A PORT GROUP INTERFACE FLAP

PR   : 1761957 [Confidential, single-source-commit] - CHANGING SPEED OF ONE INTERFACE, TRIGGER A PORT GROUP INTERFACE FLAP