Description

CPU spikes and scheduler slips are seen during the commits to modify apply-group hierarchy. The intention of the article is to provide a workaround for CPU spikes and scheduler slips seen during the commits to modify apply-group hierarchy. 

Symptoms

  • CPU Spikes and scheduler slips are observed during the commit to  modify apply-groups. Usually, issue happens, when you have large number of groups applied via apply-group hierarchy. 
  • As seen from below outputs, when there is a commit performed to modify apply-group, we see high number of apply-groups getting deleted and added back. This causes,  CPU spikes and scheduler Slips. In the below example, customer was using large number of apply-groups for defining their end customers and their services. 
  • Management Daemon causes the CPU cycles to run high . 

show configuration apply-group | display set | count<--Check for this command to check the number of apply groups currently on the box 

  [edit groups Group1]
-     interface xe-1/3/3.1000;
[edit groups]
-   Group1{
-      interfaces {
-          xe-1/3/3 {
-              unit 1505 {
-                 -                  encapsulation vlan-vpls;
-                  vlan-id 1505;
-                  family vpls {
-                      filter {
-                          input B-CONTAINMENT-TRIO;
-                      }
-                  }
-              }
-          }
-      }
-      class-of-service {
-          interfaces {
-              xe-1/3/3 {
-                  unit 1505 {
-                      classifiers {
-                          ieee-802.1 DOT1P-CLASSIFIER;
-                      }
-                      rewrite-rules {
-                          ieee-802.1 DOT1P-REWRITE;
-                      }
-                  }
-              }
-          }
-          routing-instances Test1{
-              {
-                  classifiers {
-                      exp EXP-CLASSIFIER;
-                  }
-              }
-          }
-      }
-      routing-instances LAB1{
-          {
-              
-              instance-type vpls;
-              interface xe-1/3/3.1505;
-              route-distinguisher 1.1.1.1:100;
-              vrf-target target:100:100;
-              forwarding-options {
-                  family vpls {
-                      filter {
-                          input UU-CONTAINMENT-L2CP-BASIC-TRIO;
-                      }
-                  }
-              }
-              protocols {
-                  vpls {
-                      site-range 100;
-                      mac-table-size {
-                          2500;
-                      }
-                      no-tunnel-services;
-                      site Cust1-xe-1/3/3-site8 {
-                          site-identifier 8;
-                          interface xe-1/3/3.1505;
-                      }
-                  }
-              }
-          }
-      }
-  }
[edit]
- apply-groups [ re0 re1 Cust1_-ACCT100-VRF100 Cust2_-ACCT200-VRF200 Cust3_-ACCT300-VRF300 Cust4_-ACCT400-VRF400 Cust5_-ACCT500-VRF500 Cust6_-ACCT600-VRF600 ...]

<Lines Omitted>

+ apply-groups [ re0 re1 Cust1_-ACCT100-VRF100 Cust2_-ACCT200-VRF200 Cust3_-ACCT300-VRF300 Cust4_-ACCT400-VRF400 Cust5_-ACCT500-VRF500 Cust6_-ACCT600-VRF600 ...]
 

10 root 1 171 52 0K 16K RUN ??? 42.63% idle
78049 root 1 100 0 86948K 53784K RUN 722.9H 26.61% mib2d
1712 root 2 8 -88 130M 26600K nanslp 3995.6 6.40% chassisd
2936 root 1 97 0 71652K 67532K select 797.3H 4.05% snmpd


May 19 05:32:20   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 611909 usec, system: 0 sec, 311446 usec

May 19 05:51:08   rpd[1731]: RPD_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 412400 usec, system: 0 sec, 5613 usec

May 19 06:21:38   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 424015 usec, system: 0 sec, 6909 usec

May 19 07:41:16   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 651860 usec, system: 0 sec, 259325 usec

May 19 08:02:39   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 604547 usec, system: 0 sec, 326827 usec

May 22 01:48:22   rpd[1731]: RPD_SCHED_SLIP: 8 sec scheduler slip, user: 0 sec 665738 usec, system: 0 sec, 355323 usec

May 22 05:31:19   rpd[1731]: RPD_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 413023 usec, system: 0 sec, 2981 usec

May 22 05:56:00   rpd[1731]: RPD_SCHED_SLIP: 6 sec scheduler slip, user: 0 sec 633835 usec, system: 0 sec, 303402 usec

May 22 06:41:08   rpd[1731]: RPD_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 598255 usec, system: 0 sec, 51538 usec

May 22 07:12:04   rpd[1731]: RPD_SCHED_SLIP: 7 sec scheduler slip, user: 0 sec 649392 usec, system: 0 sec, 263679 usec

May 23 05:10:37   rpd[1731]: RPD_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 553499 usec, system: 0 sec, 79190 usec

May 23 05:11:06   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 9028 usec, system: 0 sec, 254 usec

May 23 05:48:27   rpd[1731]: RPD_SCHED_SLIP: 6 sec scheduler slip, user: 0 sec 690236 usec, system: 0 sec, 136213 usec

May 23 05:48:35   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 447842 usec, system: 0 sec, 0 usec

May 23 06:32:29   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 592094 usec, system: 0 sec, 328981 usec

May 23 06:38:22   rpd[1731]: RPD_SCHED_SLIP: 4 sec scheduler slip, user: 0 sec 404597 usec, system: 0 sec, 2876 usec

May 23 06:44:05   rpd[1731]: RPD_SCHED_SLIP: 5 sec scheduler slip, user: 0 sec 421836 usec, system: 0 sec, 8143 usec

May 23 06:47:00   rpd[1731]: RPD_SCHED_SLIP: 8 sec scheduler slip, user: 0 sec 622548 usec, system: 0 sec, 267726 usec

May 23 22:12:08   rpd[1731]: RPD_SCHED_SLIP: 6 sec scheduler slip, user: 0 sec 626961 usec, system: 0 sec, 289546 usec

May 23 22:34:26   rpd[1731]: RPD_SCHED_SLIP: 6 sec scheduler slip, user: 0 sec 650559 usec, system: 0 sec, 251328 usec



 

Solution

  • Issue was root caused to large number apply-groups used to inherit configurations from groups. A workaround to this problem is to streamline or flatten the group configs and thus by having less number of apply-groups.

Modification History

NA