Description

If "ntp" and "helper" for port 123 is configured simultaneously, you may see the following error continuously, and NTP time sync may not work properly:

     "xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor"

Symptoms

A resource conflict will occur for UDP port 123 if the Junos platform is configured for both NTP Server and NTP Relay Server simultaneously.

Example:

NTP Server/Client Configuration:

set system ntp boot-server 172.27.122.250
set system ntp server 172.27.122.250


NTP Relay ("helper" for port 123) Configuration:

set forwarding-options helpers port 123 server 172.27.122.250
set forwarding-options helpers port 123 interface vlan


Symptoms:
  • If both "ntp" and "helper" for port 123 are configured simultaneously, a resource conflict will occur for the UDP port 123.
  • In the case where "ntp" is configured first and "helper” is configured later, the conflict will generate the following log message and "helper" will not work as expected.

Nov 25 13:02:47 EX8208_2_RE0 init: helper (PID 63114) started
Nov 25 13:02:51 EX8208_2_RE0 init: helper (PID 63114) exited with status=1
Nov 25 13:02:51 EX8208_2_RE0 init: helper (PID 63117) started
Nov 25 13:02:59 EX8208_2_RE0 init: helper (PID 63117) exited with status=1
Nov 25 13:02:59 EX8208_2_RE0 init: helper (PID 63118) started
Nov 25 13:03:06 EX8208_2_RE0 init: helper (PID 63118) exited with status=1
Nov 25 13:03:06 EX8208_2_RE0 init: helper (PID 63119) started
Nov 25 13:03:13 EX8208_2_RE0 init: helper (PID 63119) exited with status=1
Nov 25 13:03:13 EX8208_2_RE0 init: helper is thrashing, not restarted
These log message will stop here.

  • In the case where "helper" is configured first and "ntp” is configured later, the conflict will generate the following log message and "ntp” will not work as expected.

Nov 25 13:05:10 EX8208_2_RE0 xntpd[63799]: bind() fd 5, family 2, port 123, addr 0.0.0.0, in_classd=0 flags=8 fails: Address already in use
Nov 25 13:05:11 EX8208_2_RE0 xntpd[63799]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 25 13:05:11 EX8208_2_RE0 xntpd[63799]: sendto(172.27.122.250): Bad file descriptor
Nov 25 13:05:13 EX8208_2_RE0 xntpd[63799]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 25 13:05:25 EX8208_2_RE0 last message repeated 6 times
These log message will be generated continuously.


Solution

This happens because "xntpd" daemon for "ntp" service and "fud" daemon for "helper" service use the same UDP port 123. Junos OS does not support "ntp" and "helper" configuration for port 123 simultaneously.


Once a switch goes into the state and NTP does not work, simply removing the "helper" configuration cannot recover the state.

To recover from this state, you have to restart "xntpd" without the helper in the following order:
  1. Remove both "ntp" & "helper" for port 123.
  2. Commit.
  3. Configure "ntp" or "helper".   (Decide which of the two features to run.)
  4. Commit.
These recovery steps are shown in the recovery section of the following example:

1. Verification of problem

1.1 Helper configured for port 123
[edit]
jun@EX8208_2_RE0# set forwarding-options helpers port 123 server 172.27.122.250

[edit]
jun@EX8208_2_RE0# set forwarding-options helpers port 123 interface vlan

[edit]
jun@EX8208_2_RE0# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete

[edit]
jun@EX8208_2_RE0# exit
Exiting configuration mode

jun@EX8208_2_RE0> start shell user root
Password:
root@EX8208_2_RE0:RE:0% sockstat -l | grep 123
sockstat: struct xtcpcb size mismatch
root fud 24456 15 udp4 *:123 *:*
root tnp.sntpd 1379 4 udp4 *:6123 *:*
root@EX8208_2_RE0:RE:0%
root@EX8208_2_RE0:RE:0% exit
exit

jun@EX8208_2_RE0> configure
Entering configuration mode
Users currently editing the configuration:
jun terminal u0 (pid 6556) on since 2014-11-04 19:46:01 KST, idle 20:28:32
[edit]

1.2 NTP configured
[edit]
jun@EX8208_2_RE0# set system ntp boot-server 172.27.122.250

[edit]
jun@EX8208_2_RE0# set system ntp server 172.27.122.250

[edit]
jun@EX8208_2_RE0# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete

[edit]
jun@EX8208_2_RE0# exit
Exiting configuration mode

1.3 NTP does not work because NTP is configured together with helper for port 123
jun@EX8208_2_RE0> show ntp associations
^C[abort]

1.4 Log messages reported at this point
jun@EX8208_2_RE0> show log messages | match ntp
Nov 5 16:15:09 EX8208_2_RE0 init: ntp (PID 24909) started
Nov 5 16:15:10 EX8208_2_RE0 xntpd[24909]: ntpd 4.2.0-a Thu Jun 12 15:45:54 UTC 2014 (1)
Nov 5 16:15:10 EX8208_2_RE0 xntpd[24909]: mlockall(): Resource temporarily unavailable
Nov 5 16:15:11 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:11 EX8208_2_RE0 xntpd[24909]: sendto(172.27.122.250): Bad file descriptor
Nov 5 16:15:13 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:15 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:17 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:19 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:21 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:23 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
Nov 5 16:15:25 EX8208_2_RE0 xntpd[24909]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor

jun@EX8208_2_RE0> configure
Entering configuration mode
Users currently editing the configuration:
jun terminal u0 (pid 6556) on since 2014-11-04 19:46:01 KST, idle 20:29:39
[edit]

2. Recovery

2.1 Helper for port 123 is deleted & committed

[edit]
jun@EX8208_2_RE0# delete forwarding-options helpers

[edit]
jun@EX8208_2_RE0# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete

[edit]
jun@EX8208_2_RE0# exit
Exiting configuration mode

2.2 NTP does not work by simply removing helper for port 123, as shown in the following output where the socket state is not normal :

jun@EX8208_2_RE0> start shell user root
Password:
root@EX8208_2_RE0:RE:0% sockstat -l |grep 123
sockstat: struct xtcpcb size mismatch
root ntpd 24909 5 udp6 *:123 *:*
root ntpd 24909 6 udp4 128.0.0.1:123 *:* ? “root ntpd 25672 5 udp4 *:123 *:*” is missing
root tnp.sntpd 1379 4 udp4 *:6123 *:*
root@EX8208_2_RE0:RE:0%
root@EX8208_2_RE0:RE:0% exit
exit

jun@EX8208_2_RE0> show ntp associations
^C[abort]

jun@EX8208_2_RE0> configure
Entering configuration mode
Users currently editing the configuration:
jun terminal u0 (pid 6556) on since 2014-11-04 19:46:01 KST, idle 20:30:35
[edit]

2.3 NTP is deleted & committed.  Then NTP or Helper is reconfigured and committed. In this example, NTP is reconfigured.
[edit]
jun@EX8208_2_RE0# show | display set | match ntp
set system ntp boot-server 172.27.122.250
set system ntp server 172.27.122.250

[edit]
jun@EX8208_2_RE0# delete system ntp

[edit]
jun@EX8208_2_RE0# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete

[edit]
jun@EX8208_2_RE0# rollback 1
cload complete

[edit]
jun@EX8208_2_RE0# commit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete

2.4 NTP is verified
[edit]
jun@EX8208_2_RE0# run show ntp status
status=c011 sync_alarm, sync_unspec, 1 event, event_restart,
version="ntpd 4.2.0-a Thu Jun 12 15:45:54 UTC 2014 (1)",
processor="powerpc", system="JUNOS12.3R7.7", leap=11, stratum=16,
precision=-18, rootdelay=0.000, rootdispersion=0.120, peer=0,
refid=INIT, reftime=00000000.00000000 Thu, Feb 7 2036 15:28:16.000,
poll=4, clock=d8044d96.5c14fbd8 Wed, Nov 5 2014 16:17:42.359, state=1,
offset=0.000, frequency=-2.688, jitter=0.004, stability=0.000

[edit]
jun@EX8208_2_RE0# run show ntp associations
remote refid st t when poll reach delay offset jitter
==============================================================================
*172.27.122.250 66.129.255.62 2 - - 64 1 2.952 -2.046 0.804

[edit]
jun@EX8208_2_RE0# exit
Exiting configuration mode

2.5 Socket state is normal
jun@EX8208_2_RE0> start shell user root
Password:
root@EX8208_2_RE0:RE:0% sockstat -l | grep 123
sockstat: struct xtcpcb size mismatch
root ntpd 25672 5 udp4 *:123 *:*
root ntpd 25672 6 udp6 *:123 *:*
root ntpd 25672 7 udp4 128.0.0.1:123 *:*
root tnp.sntpd 1379 4 udp4 *:6123 *:*