This article outlines unbalanced traffic between SPC3 card vms interfaces issue and how to troubleshoot it

One of the vms interfaces has a higher CPU utilization than its counterparts. The below outputs outline one such scenario:

labroot@router> show services service-sets summary

      Service sets                                     CPU

Interface configured      Bytes used   Session bytes used  Policy bytes used  utilization

vms-4/0/0    2     3932325936 (29.98%) 639663936 ( 2.93%)   7481152 ( 0.85%)   11.83 %

vms-4/1/0    2     3928035664 (29.95%) 615220032 ( 2.82%)   7481072 ( 0.85%)   48.32 %

labroot@router> show services sessions utilization

     Session %Memory %Session-Memory Setup %Rate Drop  Teardown %CPU

Interface Count               Rate     Rate  Rate

vms-4/0/0 834746 29.98 2.92       6527          12334  10.94 Green

vms-4/1/0 798593 29.95 2.80       5514          10885  48.11 Green

To understand how load-balancing and high availability work in SPC3 and its implementation, refer to the following document:

https://www.juniper.net/documentation/us/en/software/junos/interfaces-next-gen-services/topics/concept/usf-ams-understanding.html

Unbalanced service interface utilization could be a transient issue due to mis-programming or a permanent issue due to misconfiguration. Determine if the issue is day 1 or was triggered due any changes or events [There have been instances where unexpected FPC reboot resulted in misprogramming at the PFE level, which was cleared via PIC/FPC reboot].

You may follow the below steps as part of issue isolation : 

• Verify the configuration and confirm that the traffic needs to be balanced between the ports
• Verify the rate of traffic the interfaces are processing
• Verify if high session rate and volume are contributing to high traffic rate on interfaces
•  Verify if the high volume of NAT translations is due to pool exhaustion/corruption
• Verify if the issue is due to memory leak - PR1779424, PR1618913 or any other software issue
• As all other counters like session count, setup rates etc.., seem normal, it is not clear whether the issue is due to legitimate traffic or due to traffic looping inside PIC driving the PPS and CPU high. To verify this, determine the hashing of the ingress traffic at the ingress IOC PFE. Then take a session dump on 4/1/0 via show services sessions extensive interface mams-4/1/0.
• Contact your technical support if you need assistance with the issue isolation 

Here is an elaborated version of the above bullet points with relevant command output snippets:

• For illustration purposes, we will be using a scenario where next-hop service with load-balance & redistribution feature upon member failure as follows:

  service-set sset1 {

       syslog {

          mode stream;

           source-address 192.168.0.1;

           stream syslog_sset1 {

               category nat;

               category session;

               host {

                   1.1.1.1;

                   port 5151;

                   log-tag LOG_SSET1;

               }

           }

       }

       nat-options {

           stateful-nat64 {

               clear-dont-fragment-bit;

           }

       }

       nat-rule-sets r1_src_nat64;

       nat-rule-sets r1_dst_nat64;

       next-hop-service {

           inside-service-interface ams0.3;

           outside-service-interface ams0.4;

       }

   }

   service-set sset0 {

       syslog {

           mode stream;

           source-address 192.168.0.1;

           stream syslog_sset0 {

               category nat;

               category session;

               host {

                   1.1.1.1;

                   port 5151;

                   log-tag LOG_SSET0;

               }

           }

       }

       service-set-options {

           max-sessions-per-subscriber 25000;

       }

 nat-options {

           stateful-nat64 {

               clear-dont-fragment-bit;

           }

       }

       nat-rule-sets r1_src_nat64;

       nat-rule-sets r1_dst_nat64;

       next-hop-service {

           inside-service-interface ams0.3;

           outside-service-interface ams0.4;

       }

       stateful-firewall-rules sfr;

       nat-rule-sets r0_src_napt44;

       next-hop-service {

           inside-service-interface ams0.1;

           outside-service-interface ams0.2;

       }

   } 

   ams0 {

       load-balancing-options {

           member-interface mams-4/0/0;

           member-interface mams-4/1/0;

           member-failure-options {

               redistribute-all-traffic {

                   enable-rejoin;

               }

           }

       }

       unit 1 {

           description "AMS0 IPv4 inside service interface";

           family inet {

               filter {

                   output block-frag-in;

               }

           }

           service-domain inside;

           load-balancing-options {

               hash-keys {

                   ingress-key source-ip;

               }

           }

       }

       unit 2 {

           description "AMS0 IPv4 outside service interface";

           family inet {

               filter {

                   output block-frag-out;

               }

           }

           service-domain outside;

       }

       unit 3 {

           description "AMS0 IPv6 inside service interface";

           family inet {

               filter {

                   output block-frag-in;

               }

           }

           family inet6;

           service-domain inside;

           load-balancing-options {

               hash-keys {

                   ingress-key source-ip;

               }

           }

       }

       unit 4 {

           description "AMS0 IPv6 outside service interface";

           family inet {

               filter {

                   output block-frag-out;

               }

           }

           family inet6;

           service-domain outside;

       }

   } 

• Verify the configuration and confirm that the traffic needs to be balanced between the ports as follows:

labroot@router> show interfaces load-balancing detail 

Load-balancing interfaces detail

Interface       : ams0

 State         : Up

 Last change   : 00:58:29

 Member count  : 2

 HA Model      : None

 Members       :

     Interface   Weight  State

     mams-4/0/0  10      Active

     mams-4/1/0  10      Active

• Verify the rate of traffic the interfaces are processing:

Physical interface: mams-4/0/0, Enabled, Physical link is Up

 Last flapped  : 2024-12-12 05:48:51 WIB (00:58:11 ago)

 Statistics last cleared: Never

 Traffic statistics:

  Input bytes :       3895218477236         11490753384 bps

  Output bytes :       3896272208501         11481035560 bps

  Input packets:          4314398818             1,570,878 pps

  Output packets:          4324047459             1,572,207 pps

Physical interface: mams-4/1/0, Enabled, Physical link is Up

 Last flapped  : 2024-12-12 05:48:42 WIB (00:58:20 ago)

 Statistics last cleared: Never

 Traffic statistics:

  Input bytes :      12059524950623         30184578248 bps

  Output bytes :      12060807045255         30145354792 bps

  Input packets:         42249968387             12,493,148 pps ----> The input rate is exponentially higher on 4/1/0 than 4/0/0 interface

  Output packets:         42259722244             12,493,502 pps ----> The output rate is exponentially higher on 4/1/0 than 4/0/0 interface

• Verify if high session rate and volume are contributing to high traffic rate on interfaces via the below output snippets:

labroot@router> show services sessions count ---> no significant difference in sessions count between the interfaces

Interface Service set           Valid  Invalid  Pending Other state

mams-4/0/0 sset0              838413     0     0     0

mams-4/0/0 sset1               334     0     0     0

mams-4/1/0 sset0              802800     0     0     0

mams-4/1/0 sset1               327     0     0     0

labroot@router> show services subscriber analysis ---->Not much disparity in the subscriber creation rate

 pic-name: vms-4/0/0

Subscriber Analysis Statistics:

 Total Subscribers Active       :20627

 Created Subscribers per Second    :268

 Deleted Subscribers per Second    :682

 Peak Total Subscribers Active    :21162

 Peak Created Subscribers per Second :5374

 Peak Deleted Subscribers per Second :895

 Session Drop per subscriber limit hit :6478

Subscriber Rate Data:

 Number of Samples: 3448

 pic-name: vms-4/1/0

Subscriber Analysis Statistics:

 Total Subscribers Active       :19971

 Created Subscribers per Second    :216

 Deleted Subscribers per Second    :404

 Peak Total Subscribers Active    :24478

 Peak Created Subscribers per Second :7157

 Peak Deleted Subscribers per Second :1721

 Session Drop per subscriber limit hit :0

Subscriber Rate Data:

 Number of Samples: 3449

labroot@router> show services sessions analysis

 Interface:  vms-4/0/0

Session Analysis Statistics:

 Total sessions Active                  :831907

 Total TCP Sessions Active              :668623

     Tcp sessions from gate             :0

     Tunneled TCP sessions              :0

     Regular TCP sessions               :668623

     IPv4 active Session                :668303

     IPv6 active Session                :320

 Total UDP sessions Active              :160386

     UDP sessions from gate             :0

     Tunneled UDP sessions              :0

     Regular UDP sessions               :160386

     IPv4 active Session                :160383

     IPv6 active Session                :3

 Total Other sessions Active            :2898

     IPv4 active Session                :2897

     IPv6 active Session                :1

 Created sessions per Second            :5833

 Deleted sessions per Second            :48

 Peak Total sessions Active             :840835

 Peak Total TCP sessions Active         :674029

 Peak Total UDP sessions Active         :167603

 Peak Total Other sessions Active       :3858

 Peak Created Sessions per Second       :23970

 Peak Deleted Sessions per Second       :15211

 Packets received                       :4330641000

 Packets transmitted                    :4374043623

 Slow path forward                      :18921823

 Slow path discard                      :11378845

Session Rate Data:

 Number of Samples: 3420

 Interface:  vms-4/1/0

Session Analysis Statistics:

 Total sessions Active                  :796366

 Total TCP Sessions Active              :658866

     Tcp sessions from gate             :0

     Tunneled TCP sessions              :0

     Regular TCP sessions               :658866

     IPv4 active Session                :658560

     IPv6 active Session                :306

 Total UDP sessions Active              :134517

     UDP sessions from gate             :0

     Tunneled UDP sessions              :0

     Regular UDP sessions               :134517

     IPv4 active Session                :134506

     IPv6 active Session                :11

 Total Other sessions Active            :2983

     IPv4 active Session                :2983

     IPv6 active Session                :0

 Created sessions per Second            :5355

 Deleted sessions per Second            :72

 Peak Total sessions Active             :804444

 Peak Total TCP sessions Active         :663945

 Peak Total UDP sessions Active         :137397

 Peak Total Other sessions Active       :4455

 Peak Created Sessions per Second       :31822

 Peak Deleted Sessions per Second       :22931

 Packets received                       :42320130016

 Packets transmitted                    :42359033986

 Slow path forward                      :16922847

 Slow path discard                      :11232253

Session Rate Data:

 Number of Samples: 3425

labroot@router> show services nat statistics

Interface: vms-4/0/0

Session statistics

Session statistics

 Packet Dst in NAT route               :0

 Session Created for EIF                :3693741

 Session Created for EIM                :3690316

 NAT rule lookup failures               :20998918

 NAT allocation Successes               :14121885

 NAT allocation Failures                :0

 NAT Free Successes                  :13182099

 NAT Free Failures                   :892

 NAT EIM mapping create failed             :0

 NAT EIM mapping Created                :3690316

 NAT EIM mapping Updated                :0

 NAT EIF mapping Free                 :3411954

 NAT EIM mapping Free                 :3411954

 Total Packets Processed                :2187501767

 Total Packets Forwarded                :0

 Total Packets Translated               :25460266

Interface: vms-4/1/0

Session statistics

Session statistics

 Packet Dst in NAT route               :0

 Session Created for EIF                :3949768

 Session Created for EIM                :3945713

 NAT rule lookup failures               :19081486

 NAT allocation Successes               :13910326

 NAT allocation Failures                :12573

 NAT Free Successes                  :12987142

 NAT Free Failures                   :357

 NAT EIM mapping create failed             :0

 NAT EIM mapping Created                :3945713

 NAT EIM mapping Updated                :0

 NAT EIF mapping Free                 :3675292

 NAT EIM mapping Free                 :3675292

 Total Packets Processed                :2060583417

 Total Packets Forwarded                :0

 Total Packets Translated               :4262592508 --------> More NAT translations than other interfaces

•  Verify if the high volume of NAT translations is due to pool exhaustion/corruption via:

labroot@router> show services nat source pool all

Total pools: 2

Interface: mams-4/0/0 , Service set: sset0

Pool name         : p0

Pool id           : 4

Host address base : 0.0.0.0

Port              : [1024, 65535]

Port overloading  : 1

Address assignment : no-paired

Total addresses   : 172

Translation hits  : 17540672

Ei-mapping-timeout : 120

Mapping-timeout   : 300

EIF Inbound session count: 5226

EIF Inbound session limit exceeded drops: 770580

Address range                       Ports

        36.82.96.1 - 36.82.96.170      973976

      36.82.99.253 - 36.82.99.254      11476

Total used ports  :                    985452

Error Counters:

   Out of port errors                    : 0

   Out of address errors                 : 0

   Parity port errors                    : 0

   Preserve Range errors                 : 0

   APP port allocation errors            : 0

   APP port limit allocation errors      : 0

Interface: mams-4/0/0 , Service set: sset1

Pool name         : p1

Pool id           : 5

Host address base : 0.0.0.0

Port              : [1024, 65535]

Port overloading  : 1

Address assignment : no-paired

Total addresses   : 21

Translation hits  : 53131

Ei-mapping-timeout : 300

Mapping-timeout   : 300

EIF Inbound session count: 0

EIF Inbound session limit exceeded drops: 0

Address range                       Ports

   180.242.183.129 - 180.242.183.149   383

Total used ports  :                    383

Error Counters:

   Out of port errors                    : 0

   Out of address errors                 : 0

   Parity port errors                    : 0

   Preserve Range errors                 : 0

   APP port allocation errors            : 0

   APP port limit allocation errors      : 0

Total pools: 2

Interface: mams-4/1/0 , Service set: sset0

Pool name         : p0

Pool id           : 4

Host address base : 0.0.0.0

Port              : [1024, 65535]

Port overloading  : 1

Address assignment : no-paired

Total addresses   : 170

Translation hits  : 17152690

Ei-mapping-timeout : 120

Mapping-timeout   : 300

EIF Inbound session count: 4983

EIF Inbound session limit exceeded drops: 32427

Address range                       Ports

      36.82.96.171 - 36.82.97.84       965531

Total used ports  :                    965531

Error Counters:

   Out of port errors                    : 0

   Out of address errors                 : 0 ----> In case of any errors or out-of-address error counters will increment with a corresponding log

   Parity port errors                    : 0

   Preserve Range errors                 : 0

   APP port allocation errors            : 0

   APP port limit allocation errors      : 0

Interface: mams-4/1/0 , Service set: sset1

Pool name         : p1

Pool id           : 5

Host address base : 0.0.0.0

Port              : [1024, 65535]

Port overloading  : 1

Address assignment : no-paired

Total addresses   : 21

Translation hits  : 42713

Ei-mapping-timeout : 300

Mapping-timeout   : 300

EIF Inbound session count: 0

EIF Inbound session limit exceeded drops: 0

Address range                       Ports

   180.242.183.150 - 180.242.183.170   317

Total used ports  :                    317

Error Counters:

   Out of port errors                    : 0

   Out of address errors                 : 0

   Parity port errors                    : 0

   Preserve Range errors                 : 0

   APP port allocation errors            : 0

   APP port limit allocation errors      : 0

Total pools: 2