Description

The intention of the article is to explain the sdk-vmm process which is stuck at 100% and workaround to fix it .

Symptoms

In customer box while applying a patch to both the RE’s it was observed that on backup RE the sdk-vmm process got stuck at 100%. The issue is observed when GRES is active and switchover is attempted . 

re1.PE1> show chassis routing-engine | no-more
Routing Engine status:
  Slot 0:
    Current state                  Backup
    Election priority              Master (default)
    Temperature                 33 degrees C / 91 degrees F
    DRAM                      49103 MB (49152 MB installed)
    Memory utilization           9 percent
    5 sec CPU utilization:
      User                      21 percent
      Background                 0 percent
      Kernel                    36 percent
      Interrupt                  4 percent
      Idle                      39 percent
    Model                          RE-S-2X00x6
    Start time                     2022-07-27 00:22:31 CDT
    Uptime                         629 days, 23 hours, 50 minutes, 2 seconds
    Last reboot reason             0x2000:hypervisor reboot
    Load averages:                 1 minute   5 minute  15 minute
                                       5.73       2.44       1.20
Routing Engine status:
  Slot 1:
    Current state                  Master
    Election priority              Backup (default)
    Temperature                 33 degrees C / 91 degrees F
    CPU temperature             49 degrees C / 120 degrees F
    DRAM                      49103 MB (49152 MB installed)
    Memory utilization          15 percent
    5 sec CPU utilization:
      User                      28 percent
      Background                 0 percent
      Kernel                    14 percent
      Interrupt                  6 percent
      Idle                      52 percent
    1 min CPU utilization:
      User                      10 percent
      Background                 0 percent
      Kernel                     6 percent
      Interrupt                  2 percent
      Idle                      83 percent
    5 min CPU utilization:
      User                       4 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  1 percent
      Idle                      93 percent
    15 min CPU utilization:
      User                       2 percent
      Background                 0 percent
      Kernel                     1 percent
      Interrupt                  0 percent
      Idle                      96 percent
    Model                          RE-S-2X00x6
    Serial ID                      CAPL7437
    Start time                     2022-07-27 00:00:42 CDT


show system processes extensive | except " 0.00"
last pid: 68120;  load averages:  2.12,  2.24,  1.43  up 629+23:54:21    00:16:52
317 processes: 7 running, 270 sleeping, 40 waiting

Mem: 2950M Active, 6327M Inact, 2825M Wired, 1540M Buf, 35G Free
Swap: 3072M Total, 3072M Free


  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 8515 root         103    0   727M 12244K CPU0    0  49:31 100.00% sdk-vmmd{sdk-vmmd}
67923 root         103    0  3963M  2993M CPU3    3   2:55 100.00% rpd{rpd}
   11 root         155 ki31     0K    64K RUN     0    ???  57.96% idle{idle: cpu0}
   11 root         155 ki31     0K    64K RUN     3    ???  47.36% idle{idle: cpu3}
   11 root         155 ki31     0K    64K CPU1    1    ???  43.80% idle{idle: cpu1}
   11 root         155 ki31     0K    64K RUN     2    ???  42.48% idle{idle: cpu2}
67543 root          36    0   737M 39700K select  0   0:02  12.70% mgd
   22 root         -72    -     0K   640K WAIT    1 146.3H   0.49% intr{swi1: netisr 0}
64602 root          20    0   722M 13036K uwait   1   0:33   0.39% ksyncd{ksyncd}
 8570 root          20    0   833M  4912K select  1  49.9H   0.20% ifstraced
64602 root          20    0   722M 13036K select  0   1:02   0.10% ksyncd{ksyncd}

 

Solution

The issue was related to timerfd . Timerfd is a linux standard system call that operates on interval timers . These timers are represented by file-descriptors . The timerfd had an issue which led to re-running the thread continuously spiking the CPU. To clear the issue reboot the affected RE in a MW. 

Modification History

2024-05-01 : Article Created