Article Detail

Description

This article describes the issue of Graceful Routing Engine Switchover (GRES) not supporting the configuration of a private route, such as fxp0 , when imported into a non-default instance or logical system.

Symptoms

The specific customer requirement is to have the fxp interface in a non-default instance or logical system to secure the login to the device.

For example, All the static routes, including fxp0 , in the default instance have been imported into the TO_ABC_VPN_s routing instance:

> show configuration routing-options static rib-group TO_ABC_VPN_s; ….. TO_ABC_VPN_s { import-rib [ inet.0 TO_ABC_VPN_s.inet.0 ]; }

{MASTER}

> run show route table TO_ABC_VPN_s.inet.0 TO_ABC_VPN_s.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 1.1.1.1/32 *[Static/5] 00:02:12 > to 192.168.1.2 via ge-1/0/0.0 10.0.0.0/8 *[Static/5] 00:02:12 > to 172.27.100.1 via fxp0.0 172.16.0.0/12 *[Static/5] 00:02:12 > to 172.27.100.1 via fxp0.0 192.168.0.0/16 *[Static/5] 00:02:12 > to 172.27.100.1 via fxp0.0

{MASTER}

The status of GRES reports the Kernel database: Connection error, Initialize error error message, after the RE mastership switchover occurs:

> request chassis routing-engine master switch Toggle mastership between routing engines ? [yes,no] (no) yes Resolving mastership... Complete. The other routing engine becomes the master.

{BACKUP}
user@m320-re0> show system switchover Graceful switchover: On Configuration database: Ready Kernel database: Connection error, Initialize error Peer state: Steady State

{BACKUP}

As per the ksyncd log on the backup RE, this issue is related to rtb TO_ARBOR_VPN_s :

> show log ksyncd | last Dec 05 10:15:44 Not runnable attempt 0 reason: hard error (errors: init_err connection_errsoft_mask_err) Dec 05 10:15:44 closing connection to master Dec 05 10:15:44 cleaning up kernel state Dec 05 10:15:53 route_table_walk: af 55 initial rtb get failed: Protocol not supported Dec 05 10:15:53 route_table_walk: af 56 initial rtb get failed: Protocol not supported Dec 05 10:15:53 route_table_walk: af 63 initial rtb get failed: Protocol not supported Dec 05 10:15:53 route_table_walk: af 64 initial rtb get failed: Protocol not supported Dec 05 10:15:53 route_table_walk: af 67 initial rtb get failed: Protocol not supported Dec 05 10:15:53 vlan getnext failed for idx 0: Invalid argument Dec 05 10:15:53 delete rtb TO_ABC_VPN_ s af 2 tid 7: No such file or directory Dec 04 10:15:53 route_table_walk: af 55 initial rtb get failed: Protocol not supported Dec 04 10:15:53 route_table_walk: af 56 initial rtb get failed: Protocol not supported Dec 04 10:15:53 route_table_walk: af 63 initial rtb get failed: Protocol not supported Dec 04 10:15:53 route_table_walk: af 64 initial rtb get failed: Protocol not supported Dec 04 10:15:53 route_table_walk: af 67 initial rtb get failed: Protocol not supported Dec 04 10:15:53 fc_fabric getnext failedfor idx 0: Invalid argument Dec 04 10:15:53 fmgrp get failed for idx 0: Invalid argument Dec 04 10:15:53 Successfully cleaned NOTIFY ifstates Dec 04 10:15:53 check_nh() failed: public rnh idx 580 af 2 nhtype mcst Dec 04 10:15:53 check_nh() failed: public rnh idx 581 af 2 nhtype bcst Dec 04 10:15:53 check_nh() failed: public rnh idx 582 af 2 nhtype recv Dec 04 10:15:53 check_nh() failed: public rnh idx 583 af 2 nhtype dscd Dec 04 10:15:53 check_nh() failed: public rnh idx 584 af 2 nhtype mdsc Dec 04 10:15:53 check_nh() failed: public rnh idx 585 af 2 nhtype rjct Dec 04 10:15:53 check_nh() failed: public rnh idx 586 af 2 nhtype locl Dec 04 10:15:53 check_nh() failed: public rnh idx 587 af 2 nhtype deny Dec 04 10:15:53 check_nh() failed: public rnh idx 588 af 2 nhtype dscd Dec 04 10:15:53 Nexthop check found 9 public nexthops Dec 04 10:15:54 check_nh() failed: public rnh idx 580 af 2 nhtype mcst Dec 04 10:15:54 check_nh() failed: public rnh idx 581 af 2 nhtype bcst Dec 04 10:15:54 check_nh() failed: public rnh idx 582 af 2 nhtype recv Dec 04 10:15:54 check_nh() failed: public rnh idx 583 af 2 nhtype dscd Dec 04 10:15:54 check_nh() failed: public rnh idx 584 af 2 nhtype mdsc Dec 04 10:15:54 check_nh() failed: public rnh idx 585 af 2 nhtype rjct Dec 04 10:15:54 check_nh() failed: public rnh idx 586 af 2 nhtype locl Dec 04 10:15:54 check_nh() failed: public rnh idx 587 af 2 nhtype deny Dec 04 10:15:54 check_nh() failed: public rnh idx 588 af 2 nhtype dscd Dec 04 10:15:54 Nexthop check found 9 public nexthops Dec 04 10:15:54 cleanup complete 0 Dec 04 10:15:54 ksyncd_set_slave_gres_ready_state: vks id 0, Setting hw.re.is_slave_peer_gres_ready: 0

The de-activation or activation of GRES does not help. The only recovery method is rebooting the backup RE to temporarily fix the faulty GRES status. This issue will re-occur if an RE switchover is performed later.

Solution

The fxp0 routes that exist in the the TO_ABC_VPN_s routing instance are created as public routes.
When GRES replication is triggered, ksyncd tries to perform the cleanup of public routes and public tables. However, it does not delete neither private routes nor private Route Tables.
Now, when it tries to delete the public routes and public table for the TO_ABC_VPN_s routing instance, it fails as it had previously skipped the deletion of private route entries in the default instance.
GRES gets into this inconsistent state and reports the Kernel database: Connection error, Initialize error error message when an RE switchover is executed.

GRES does not support the configuration, when the private route, such as fxp0 , is imported to a non-default instance or logical system.

In certain cases, the fxp0 route can be filtered from being imported. To do so, you can apply a route policy:

{MASTER}[edit policy-options policy-statement exclude-fxp0]
# show
term 1 {
   from interface fxp0.0;
   then reject;
}
term 2 {
   then accept;
}

{MASTER}[edit routing-options rib-groups]
# show
TO_ABC_VPN_s {
      import-rib [ inet.0 TO_ABC_VPN_s.inet.0 ];
      import-policy exclude-fxp0;
}

Check the TO_ABC_VPN_s.inet.0 route table, after applying the policy; fxp0 routes are no longer present:

# run show route table TO_ABC_VPN_s.inet.0

TO_ABC_VPN_s.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32 *[Static/5] 00:00:05
         > to 192.168.1.2 via ge-1/0/0.0

{MASTER}[edit routing-options]

When the RE switchover is verified, the issue does not re-occur:

> request chassis routing-engine master switch
Toggle mastership between routing engines ? [yes,no] (no) yes

Resolving mastership...
Complete. The other routing engine becomes the master.

{BACKUP}
user@m320-re0> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Synchronizing
Peer state: Steady State

{BACKUP}
user@m320-re0> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Ready
Peer state: Steady State

{BACKUP}