Description

 If the error message "kern.maxfiles limit exceeded by uid 0, please see tuning(7)" is seen on the console prompt of a QFX3500, users will not be able to log in or perform any activities on the switch. 

Symptoms

On a newly installed QFX3500 the user is unable to log into the QFX3500 and the following message is displayed at the console prompt:


"kern.maxfiles limit exceeded by uid 0, please see tuning(7)"

  • This is caused by the total number of open files on the system exceeding the maximum limit.
  • The problem can be temporarily resolved by rebooting the system. However, the number of open files will grow over time (at the rate of 1 open file every 4 sec), so the issue will reappear after approximately 8 to 10 hours.
  • An upgrade/downgrade will not resolve the issue either.
  • The workaround at the end of this KB resolves the issue permanently.
  • This issue is being addressed through PR-871506

Solution

  • The KDM (Kernel Diagnostics Managment) logic was introduced in the QFX3500 for better management of kernel panic events.
  • This logic in Junos polls for the existence of core files in the swap/dump area by checking a specific field which would indicate the state of kernel dumps.
  • The value can be corrupted on a new devices unless there was a valid core dump in the history of the box or if the dump was cleared by copying to the file system.
  • The corrupted/illegal value causes the system to believe that there is a valid core dump.
  • This eventually affects the kdm process and causes the open file directory to never close, eventually building up the number of open files on the system and making it unstable.

 How to identify this condition and correct it:


1. From shell, check the number of open files on system by using the command "sysctl -a | grep files"

  • The output of this command will look something like this:

root@QFX1% sysctl -a | grep files
kern.maxfiles: 12328
kern.maxfilesperproc: 12328
kern.openfiles: 585                                   >>>>>>>>>>>>>>>>> indicates the number of kern.openfiles
p1003_1b.mapped_files: 1
root@QFX1%

NOTE:  If the system is affected by this issue, there will be one file opened every 4 seconds. Hence, you will see the variable ‘kern.openfiles’ increasing significantly over time.

2. Check the output of "fstat" command from the shell prompt. 
  • The "fstat" command essentially provides the file status of a system running FreeBSD. By checking the output of this command, we can determine if the system is exhibiting this issue.
  • First, save the output of fstat to a file.
root@QFX1% fstat >> /var/tmp/fstat.out
  • You can then check the number of times kdm_savekcore process has opened the da0s3b partition.
root@QFX1% cat /var/tmp/fstat.out | grep kdm_savekcore
  • On a system that IS experiencing this issue and is in a problem state, the following output will be seen:
root kdm_savekcore 70 0 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 1 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 2 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 3 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 4 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 5 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 6 /dev 68 crw-r----- da0s3b rw\par
root kdm_savekcore 70 7 /dev 68 crw-r----- da0s3b rw\par

NOTE:  On a system that IS NOT experiencing an issue, there should be no files opened by this process so your output will be null.

How to correct this condition using the 'savecore -c' command:

  • Running the command ‘savecore –c’ (savecore hyphen lowercase c) sets the flag to ‘ cleared kernel dump’ and thus avoid any further processing which causes the issue.  The issue will not be seen again after this flag is cleared. This command will clear the flag and prevent the kdm process from opening new files.
  • To run this command, enter the shell prompt on the QFX3500:
root@QFX1% savecore –c
  • After you run the 'savecore -c' command it is highly recommended that you reboot the system to clear the existing open files and reset the system