Article Detail

Description

This article describes about the auditd process crash when accounting Radius Server is not reachable.

Symptoms

The audit process (auditd) gathers information about system accounting events from other system processes and sends the information to RADIUS servers in an Accounting-Request message. After failing to receive a response from the indicated server, auditd waits for a timeout period and resends the message.

During this event, we observe AUDITD_RADIUS_REQ_TIMED_OUT messages sent to the Radius servers getting timed out.

Following logs will be observed during the event:

Jan 29 03:52:08.464 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 03:52:11.464 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 03:52:14.465 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.172(10.92.184.172).1813

Jan 29 03:52:20.466 2023  fw1 last message repeated 2 times

Jan 29 03:52:23.467 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.173(10.92.184.173).1813

Jan 29 03:52:29.468 2023 fw1 last message repeated 2 times

Jan 29 03:52:32.468 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.172(10.23.254.172).1813

Jan 29 03:52:38.470 2023  fw1 last message repeated 2 times

Jan 29 03:52:41.470 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_DROPPED: Discarded Accounting-Request message; no RADIUS server responded
 
 
Jan 29 03:52:44.471 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.172(10.23.254.172).1813

Jan 29 03:52:47.471 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.172(10.23.254.172).1813

Jan 29 03:52:50.472 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 03:52:53.472 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 03:52:56.473 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 03:52:59.474 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.172(10.92.184.172).1813

Jan 29 03:53:05.475 2023 fw1 last message repeated 2 times

Jan 29 03:53:08.475 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.173(10.92.184.173).1813

Jan 29 03:53:14.476 2023 fw1 last message repeated 2 times

Jan 29 03:53:17.477 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_DROPPED: Discarded Accounting-Request message; no RADIUS server responded

Jan 29 04:00:00.178 2023 fw1 inetd[1192]: /usr/sbin/sshd[229]: exited, status 255

Jan 29 04:00:01.319 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: 
Retransmitted request to RADIUS server 10.92.184.173(10.92.184.173).1813

Jan 29 04:00:04.320 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.173(10.92.184.173).1813

Jan 29 04:00:07.320 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.172(10.23.254.172).1813

Jan 29 04:00:13.322 2023  fw1 last message repeated 2 times

Jan 29 04:00:16.322 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.23.254.174(10.23.254.174).1813

Jan 29 04:00:22.323 2023 fw1 last message repeated 2 times

Jan 29 04:00:25.324 2023  fw1 auditd[12802]: AUDITD_RADIUS_REQ_TIMED_OUT: Retransmitted request to RADIUS server 10.92.184.172(10.92.184.172).1813

Jan 29 04:00:31.325 2023 fw1 last message repeated 2 times

Jan 29 04:00:34.326 2023 fw1 auditd[12802]: AUDITD_RADIUS_REQ_DROPPED: Discarded Accounting-Request message; no RADIUS server responded

The RADIUS servers are not reachable and AUDITD_RADIUS_REQ_TIMED_OUT logs are continuously seen.
Junos device is trying to send the accounting request packet to accounting server but failed.
Junos device will send request and wait for sometime to receive the response. If the response is not received, timeout happens and auditd_rad_timeout_handler gets called.
The requests gets piled up leading to evSelectFD function failure which triggers the core.

Below core dumps will be generated:

-rw-rw----  1 root  wheel     886196 Jan 30 03:54 /var/tmp/auditd.core-tarball.4.tgz
-rw-rw----  1 root  wheel      52594 Jan 30 03:53 /var/tmp/auditd.core.4.gz

Solution

The cause of the AUDITD log messages are due to the RADIUS server not responding.

Possible reasons include
(a) the server machine is down, too busy, or unreachable
(b) the server application is down or too busy
(c) the shared RADIUS secret sent by auditd does not match the secret on the server.

This issue will not occur if the accounting radius server is responding.

Kindly check why the accounting server is unreachable and ensure the connectivity to accounting radius server.

Description

Symptoms

Solution

Modification History