When Qfabric does not list one of the data plane interfaces, we need to bring the interface back up and join the fabric topology.
When Qfabric fabric topology misses an IC and RSNG fte incoming/outgoing link, the IC end's link status is down. Also, the port receives power from the remote end as shown below.
Qfabric did not list RSNG22-2 and IC-1 incoming/outgoing links while checking the 'show chassis fabric connectivity' output. Link fte-3/0/5 from IC1 to fte-0/1/0 on RSNG22-2
'show chassis fabric connectivity'
user@switch> show chassis fabric connectivity device qf1-rsng22-2 Device ID: qf1-rsng22-2, Type: qfx5100-48s-6qf
Fabric: Incoming links: qf1-ic2:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/1 Ok qf1-ic3:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/2 Ok qf1-ic4:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/3 Ok Fabric: Outgoing links: qf1-rsng22-2:fte-0/1/1 -> qf1-ic2:fte-3/0/5 Ok qf1-rsng22-2:fte-0/1/2 -> qf1-ic3:fte-3/0/5 Ok qf1-rsng22-2:fte-0/1/3 -> qf1-ic4:fte-3/0/5 Ok
Replacing L1 units SFP and cable did not recover.
RSNG22 shows interface is up whereas IC1 shows interface admin is up, but the link down:
RSNG22:
fte-1/1/0 up up fte-1/1/0.32768 up up eth-switch
fte-3/0/5 up down fte-3/0/5.32768 up down eth-switch
'show interfaces diagnostics optics fte-3/0/5'
user@switch-ic1> show interfaces diagnostics optics fte-3/0/5 Physical interface: fte-3/0/5 Module temperature : 29 degrees C / 83 degrees F Module voltage : 3.2780 V Module temperature high alarm threshold : 0 degrees C / 32 degrees F Module temperature low alarm threshold : 0 degrees C / 32 degrees F Module temperature high warning threshold : 0 degrees C / 32 degrees F Module temperature low warning threshold : 0 degrees C / 32 degrees F Module voltage high alarm threshold : 0.0000 V Module voltage low alarm threshold : 0.0000 V Module voltage high warning threshold : 0.0000 V Module voltage low warning threshold : 0.0000 V Laser bias current high alarm threshold : 0.000 mA Laser bias current low alarm threshold : 0.000 mA Laser bias current high warning threshold : 0.000 mA Laser bias current low warning threshold : 0.000 mA Laser output power high alarm threshold : 0.0000 mW / - Inf dBm Laser output power low alarm threshold : 0.0000 mW / - Inf dBm Laser output power high warning threshold : 0.0000 mW / - Inf dBm Laser output power low warning threshold : 0.0000 mW / - Inf dBm Laser rx power high alarm threshold : 0.0000 mW / - Inf dBm Laser rx power low alarm threshold : 0.0000 mW / - Inf dBm Laser rx power high warning threshold : 0.0000 mW / - Inf dBm Laser rx power low warning threshold : 0.0000 mW / - Inf dBm Lane 0 Laser bias current : 6.558 mA Laser output power : 0.000 mW / - Inf dBm Laser receiver power : 0.664 mW / -1.78 dBm Laser bias current high alarm : Off Laser bias current low alarm : Off Laser bias current high warning : Off Laser bias current low warning : Off Laser receiver power high alarm : Off Laser receiver power low alarm : Off Laser receiver power high warning : Off Laser receiver power low warning : Off Tx loss of signal functionality alarm : Off Rx loss of signal alarm : Off Lane 1 Laser bias current : 6.693 mA Laser output power : 0.000 mW / - Inf dBm Laser receiver power : 0.659 mW / -1.81 dBm Laser bias current high alarm : Off Laser bias current low alarm : Off Laser bias current high warning : Off Laser bias current low warning : Off Laser receiver power high alarm : Off Laser receiver power low alarm : Off Laser receiver power high warning : Off Laser receiver power low warning : Off Tx loss of signal functionality alarm : Off Rx loss of signal alarm : Off Lane 2 Laser bias current : 6.751 mA Laser output power : 0.000 mW / - Inf dBm Laser receiver power : 0.699 mW / -1.56 dBm Laser bias current high alarm : Off Laser bias current low alarm : Off Laser bias current high warning : Off Laser bias current low warning : Off Laser receiver power high alarm : Off Laser receiver power low alarm : Off Laser receiver power high warning : Off Laser receiver power low warning : Off Tx loss of signal functionality alarm : Off Rx loss of signal alarm : Off Lane 3 Laser bias current : 6.705 mA Laser output power : 0.000 mW / - Inf dBm Laser receiver power : 0.702 mW / -1.53 dBm Laser bias current high alarm : Off Laser bias current low alarm : Off Laser bias current high warning : Off Laser bias current low warning : Off Laser receiver power high alarm : Off Laser receiver power low alarm : Off Laser receiver power high warning : Off Laser receiver power low warning : Off Tx loss of signal functionality alarm : Off Rx loss of signal alarm : Off
The pafxpc process running for that particular FPC[fpc3] restarting on IC recovers from this issue.
Find the corresponding pafxpc process for the FPC3 from Qfabric IC:
user@switch-ic1% ps aux | grep /usr/sbin/pafxpc root 2250 50.3 3.9 287664 130976 ?? S 8Sep18 771943:40.47 /usr/sbin/pafxpc -s 3 <-- For FPC3, the number indicates LC slot number. root 2249 37.5 3.9 287664 130976 ?? L 8Sep18 592154:30.14 /usr/sbin/pafxpc -s 2 root 2247 36.8 3.9 287664 130976 ?? L 8Sep18 607134:40.64 /usr/sbin/pafxpc -s 0 root 2248 36.8 3.9 287664 130976 ?? R 8Sep18 605553:07.50 /usr/sbin/pafxpc -s 1 root 2251 35.3 3.9 287664 130976 ?? S 8Sep18 604933:26.10 /usr/sbin/pafxpc -s 4 root 2252 35.3 3.9 287664 130976 ?? R 8Sep18 586400:51.32 /usr/sbin/pafxpc -s 5 root 2253 33.1 3.9 287664 130976 ?? L 8Sep18 560371:50.37 /usr/sbin/pafxpc -s 6 root 2258 21.8 2.6 256912 86832 ?? R 8Sep18 349982:08.05 /usr/sbin/pafxpc -s 11 root 2255 21.0 2.6 256912 86760 ?? R 8Sep18 351239:42.01 /usr/sbin/pafxpc -s 8 root 2257 21.0 2.6 256912 86824 ?? R 8Sep18 353067:55.56 /usr/sbin/pafxpc -s 10 root 2263 21.0 2.6 256912 86820 ?? R 8Sep18 354581:59.96 /usr/sbin/pafxpc -s 15 root 2259 20.3 2.6 256912 86820 ?? R 8Sep18 353144:37.69 /usr/sbin/pafxpc -s 12 root 2261 20.3 2.6 256912 86776 ?? R 8Sep18 353940:15.46 /usr/sbin/pafxpc -s 14 root 2256 19.5 2.6 256912 86820 ?? R 8Sep18 352473:28.94 /usr/sbin/pafxpc -s 9 root 2260 18.0 2.6 256912 86780 ?? R 8Sep18 354550:15.57 /usr/sbin/pafxpc -s 13 root 92333 3.0 0.0 3388 1496 p0 S+ 2:34PM 0:00.03 grep /usr/sbin/pafxpc root 2254 0.0 1.6 54060 53964 ?? S 8Sep18 261:23.72 /usr/sbin/pafxpc -s 7
Kill and check the process is re-spawned:
user@switch-ic1% ps aux | grep /usr/sbin/pafxpc root 2250 50.3 3.9 287664 130976 ?? R 8Sep18 773686:49.74 /usr/sbin/pafxpc -s 3 user@switch-ic1% kill -9 2250 user@switch-ic1% ps aux | grep /usr/sbin/pafxpc root 73995 28.6 3.9 287664 130976 ?? L 8:27PM 0:04.27 /usr/sbin/pafxpc -s 3
The interface IC link status is up after the pafxpc restart:
fte-3/0/5 up up fte-3/0/5.32768 up up eth-switch
The interface is now listed in the fabric connectivity output as well:
user@switch> show chassis fabric connectivity device qf1-rsng22-2 May 27 20:34:17 Device ID: qf1-rsng22-2, Type: qfx5100-48s-6qf Fabric: Incoming links: qf1-ic1:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/0 Ok qf1-ic2:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/1 Ok qf1-ic3:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/2 Ok qf1-ic4:fte-3/0/5 -> qf1-rsng22-2:fte-0/1/3 Ok Fabric: Outgoing links: qf1-rsng22-2:fte-0/1/0 -> qf1-ic1:fte-3/0/5 Ok qf1-rsng22-2:fte-0/1/1 -> qf1-ic2:fte-3/0/5 Ok qf1-rsng22-2:fte-0/1/2 -> qf1-ic3:fte-3/0/5 Ok qf1-rsng22-2:fte-0/1/3 -> qf1-ic4:fte-3/0/5 Ok