EX2300 Disconnected from MIST After Connect IPTV System

EX2300 switch:

ge-0/0/0 to gateway switch (192.168.100.250) << path to internet

ge-0/0/22 to multicast receiver client (192.168.100.253)

ge-0/0/23 to multicast source (192.168.2.136)

All ports are configured as access ports in the same VLAN New_IPTV id 3XXX.

After testing customer provided the following test result:

1. Multicast client is getting video streaming from server normally.

2. EX2300 went disconnected in MIST portal

 - we can see ping response from client to this EX2300 is very high (1,000 ms) after we connect multicast source to EX2300. After few minutes Mist will see this EX2300 down.

JTAC lab tests:

I tried to replay customer traffic sent to destination IP 224.2.2.21 in my lab and found similar problem when there is more than 500 PPS hitting the CPU queue.  If I reduce the number of packets to less than 400 PPS, I can see the ping to IRB.300 works fine without any issues. Hence the idea was to make sure these multicast packets are not punt to CPU and suggested specific ways to avoid them hitting CPU. There was lot of traffic hitting the CPU "queue 3" which is rate limited to 500 pps. This queue is used for both unregistered multicast traffic and also for ICMP ( ping ). Queue number 3 is used for multiple protocol traffic including:

PROTOCOL             EX2300 Queue No. PPS

SSH, SNMP, Local Control IP, PING     3     500

TFXPC0(IPTV02 vty)# show halp-pkt pkt-stats  

PKT TX STATS:

tx_req       : 89335

tx_success      : 89335

tx_lazy_vlan_tag_token_get_fail : 2074

tx_l2cookie_found    : 8

tx_l2token_lookup_success  : 8

tx_arr_req      : 89114

tx_arr_success     : 89114

tx_with_tx_pbmp    : 89273

tx_with_no_tx_pbmp    : 62

tx_pkt_proc_nobypass_mode  : 62

tx_pkt_proc_bypass_mode  : 89273

tx_pkt_reinsert_bcast   : 62

PKT RX STATS:

rx_req       : 40846

rx_success      : 40846

rx_buf_alloc     : 41792

rx_pkt_duplicated    : 11

rx_pkt_dfw_cloned    : 40846

rx_pkt_ipv4     : 39584

rx_pkt_igmp_report    : 11

rx_pkt_mc_unreg_enqueued  : 38805. <<<<<<<<<<<<

rx_pkt_vxlan_not_found_encap_pkt : 79423

rx_pkt_mc_throttle_drop  : 8030

user@switch> ping 192.168.100.251 

PING 192.168.100.251 (192.168.100.251): 56 data bytes

<snip>

64 bytes from 192.168.100.251: icmp_seq=5169 ttl=64 time=1286.474 ms

64 bytes from 192.168.100.251: icmp_seq=5170 ttl=64 time=1282.033 ms

64 bytes from 192.168.100.251: icmp_seq=5171 ttl=64 time=1283.381 ms

64 bytes from 192.168.100.251: icmp_seq=5295 ttl=64 time=1286.323 ms

64 bytes from 192.168.100.251: icmp_seq=5296 ttl=64 time=1280.616 ms

64 bytes from 192.168.100.251: icmp_seq=5297 ttl=64 time=1289.968 ms

--- 192.168.100.251 ping statistics ---

5336 packets transmitted, 163 packets received, 96% packet loss

round-trip min/avg/max/stddev = 1278.429/1286.436/1363.349/11.436 ms

{master:0}

user@switch>

Solution:

=========

To stop the unregistered multicast traffic was getting trapped to CPU: Have a multicast receiver always connected directly on the switch for encoder traffic. Also the switch was trying to resolve the source IP of encoder as IRB was configured, suggested to remove the IRB.3000 or suggested to have the source IP of the multicast traffic changed to an IP within vlan subnet of the IRB.