This article discusses about the possible cause of "general-authentication-service subsystem is not running" message while checking PCRF state.
This issue was seen on a subscriber management router.
Customer reported that they are unable to fetch PCRF state and getting below error message:
labroot@mx960> show network-access pcrf state
error: the general-authentication-service subsystem is not running
Observed that the authd process was in OFFLINE state:
labroot@mx960> show shm-ipc statistics
Oct 15 09:57:48
SHM-IPC User Statistics:
Name: authd
Role: PRODUCER, Status: OFFLINE
Performance counters
Reads Writes Errors
Socket Datagrams Socket Datagrams Socket Datagrams
0 20 0 12 0 0
Producer message counters
Messages Responses received Backpressure
Sent ACKs NAKs Count
2296675 2296665 10 0
Connection to bbe-smgd, Status OPEN
Messages posted 1279972, pending 0, max pending 56, held 0, complete 1279972
labroot@mx960> show log messages | match authd | last 20
Oct 15 09:58:11
Oct 15 09:56:03 mx960_RE0 authd[77267]: dc=0 ts=1 cd=5 rmin=30 rn=30 ri=10 lc=0 la=90 cc=0 rc=0 ct=250000 nc=0 ml=409600 ns=409600 ss=1728986163
Oct 15 09:56:04 mx960_RE0 authd[77267]: LIBJSNMP_SA_IPC_REG_ROWS: ns_subagent_register_mibs: registering 11 rows
Oct 15 09:56:04 mx960_RE0 authd[77267]: Startup: ISSU State is IDLE
Oct 15 09:56:04 mx960_RE0 kernel: rts_ifstate_chk_multi_registration: daemon authd(77267) has previously registered 1 time(s)
Oct 15 09:56:04 mx960_RE0 authd[77267]: AUTHD_ORIGIN_STATE_ID: Origin State Id is set to 1662008447 secs from sdb
Oct 15 09:56:06 mx960_RE0 kernel: vm_fault: pager read error, pid 77267 (authd)
Oct 15 09:56:07 mx960_RE0 cpcdd[19855]: authConnect ../../../../../../src/junos/usr.sbin/cpcdd/cpcddAuthSession.cc:186: could not connect to authd, error = failure; will retry
Oct 15 09:56:12 mx960_RE0 cpcdd[19855]: authConnect ../../../../../../src/junos/usr.sbin/cpcdd/cpcddAuthSession.cc:186: could not connect to authd, error = failure; will retry
Oct 15 09:56:17 mx960_RE0 cpcdd[19855]: authConnect ../../../../../../src/junos/usr.sbin/cpcdd/cpcddAuthSession.cc:186: could not connect to authd, error = failure; will retry
Observed that the authd process was continuously crashing.
We tried to restart the general-authentication-service using the "restart general-authentication-service" command, however, the authd process kept on crashing with core-dump:
labroot@mx960_RE0> restart general-authentication-service
error: General authentication process is not running
General authentication process started, pid 77217
labroot@mx960_RE0> show system processes extensive | match authd
77217 root 93 0 730M 376M CPU0 0 0:13 69.87% authd
77217 root 96 0 730M 409M RUN 1 0:16 80.96% authd
77239 root 26 0 434M 58456K CPU2 2 0:00 1.17% authd
77239 root 78 0 726M 292M CPU3 3 0:02 19.87% authd
77239 root 82 0 730M 296M CPU0 0 0:05 34.86% authd
Oct 15 09:56:14
77267 root 92 0 730M 373M CPU3 3 0:11 64.26% authd
Oct 15 09:56:19
77267 root 95 0 730M 389M CPU0 0 0:15 75.39% authd
-rw------- 1 root wheel 26109206 Oct 15 09:55 /var/tmp/authd.core.0.gz
-rw------- 1 root wheel 6012928 Oct 15 09:55 /var/tmp/authd.core.1.gz
-rw------- 1 root wheel 26249735 Oct 15 08:32 /var/tmp/authd.core.2.gz
-rw------- 1 root wheel 26147524 Oct 15 08:32 /var/tmp/authd.core.3.gz
-rw------- 1 root wheel 26154551 Oct 15 08:33 /var/tmp/authd.core.4.gz
From the logs, we found that the master RE0 was in a bad state:
Oct 15 10:08:31 mx960_RE0 kernel: md20.uzip: UZIP(zlib) inflate() failed
Oct 15 10:08:31 mx960_RE0 kernel: g_vfs_done():md20.uzip[READ(offset=251854848, length=2048)]
Oct 15 10:08:31 mx960_RE0 kernel: vm_fault: pager read error, pid 77415 (ifinfo)
Oct 15 10:08:31 mx960_RE0 kernel: g_vfs_done():md20.uzip[READ(offset=257609728, length=2048)]
Oct 15 10:08:51 mx960_RE0 kernel: md20.uzip: UZIP(zlib) inflate() failed
Oct 15 10:08:51 mx960_RE0 kernel: g_vfs_done():md20.uzip[READ(offset=251854848, length=2048)]error = 86
Oct 15 10:08:51 mx960_RE0 kernel: vm_fault: pager read error, pid 77425 (ifinfo)
Oct 15 10:08:51 mx960_RE0 kernel: g_vfs_done():
These messages are seen due to kernel read errors. This is related to the underlying FreeBSD based kernel unable to read content from the hard disk and hence can be either the filesystem or kernel issue.
Please refer KB36723 [juniper.net] for more information about this error message.
We performed a mastership switchover and rebooted the RE0. Please note that the mastership switchover in such condition would be an ungraceful one as the current master RE is in a bad state. The ungraceful mastership switchover will cause all FPCs to restart.
After RE switchover , need to confirm all subscribers are up on new master RE, if we still see any uncertainties, we might need to reboot the whole system to have a clean state of the subscriber database.
While RE restarts, file system check (FSCK) automatically runs and repairs the file system. We need to monitor the console logs and confirm FSCK is completed correctly.
Once the RE comes online, monitor if any "vm_fault" messages are still appearing or not.
In the RE does not recover from reboot or "vm_fault" messages appear again, contact JTAC for further assistance.