Article Detail

Description

A router upgraded from 21.4 to 22.4 is now showing alarms tied to task replication:

user@router-re0> show chassis alarms
4 alarms currently active
Alarm time Class Description
2025-04-30 12:53:45 PDT Minor "Replication error for protocol LDP"
2025-04-30 12:48:45 PDT Minor "Replication error for protocol RSVP"
2025-04-30 12:48:45 PDT Minor "Replication error for protocol MPLS"
2025-04-30 12:48:45 PDT Minor "Replication error for protocol BGP"

However, all state is synched on the Master RE:

user@router-re0> show task replication
Stateful Replication: Enabled
RE mode: Master

Protocol Synchronization Status
OSPF Complete
BGP Complete
MPLS Complete
RSVP Complete
LDP Complete

{master}
user@router-re0> show database-replication summary

General:
Graceful Restart Enabled
Mastership Master
Connection Up
Database Synchronized
Message Queue Ready

And, the backup is clean as well:

{backup}
user@router-re1> show system switchover
Graceful switchover: On
Configuration database: Ready
Kernel database: Ready
Switchover Status: Ready

Symptoms

Below listed alarms observed after upgrade.

Solution

The alarms are generated by a slax script installed on the router with the following event-options:

set event-options generate-event 5MIN time-interval 300
set event-options policy p1 events KERNEL
set event-options policy p1 attributes-match KERNEL.message matches KERN_ARP_DUPLICATE_ADDR
set event-options policy p1 then raise-trap
set event-options policy Memory_Capacity events /mfs
set event-options policy Memory_Capacity then raise-trap
set event-options policy FPC-issue events SYSTEM
set event-options policy FPC-issue attributes-match SYSTEM.message matches "alarm set, FPC"
set event-options policy FPC-issue then raise-trap
set event-options policy MD5_Digest events tcp_auth_ok
set event-options policy MD5_Digest then raise-trap
set event-options policy task-replication-check events 5MIN
set event-options policy task-replication-check then event-script TaskReplication.slax
set event-options policy task-replication-TRAP1 events SYSTEM
set event-options policy task-replication-TRAP1 attributes-match SYSTEM.message matches "database-replication state is not Synchronized"
set event-options policy task-replication-TRAP1 then raise-trap
set event-options policy task-replication-TRAP2 events SYSTEM
set event-options policy task-replication-TRAP2 attributes-match SYSTEM.message matches "Replication error for protocol"
set event-options policy task-replication-TRAP2 then raise-trap
set event-options event-script optional
set event-options event-script file TaskReplication.slax

The script is no longer able to parse the command output due to changes in output format in upgraded Junos release.

A new version of the script has been created, listed below. With this script, the output format is changed and the errors will no longer be seen:

/*

Licensed under the Juniper Networks Script Software License (the "License").

You may not use this script file except in compliance with the License, which is located at

http://www.juniper.net/support/legal/scriptlicense/

Unless required by applicable law or otherwise agreed to in writing by the parties,

software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES

OR CONDITIONS OF ANY KIND, either express or implied.

# Description

This script periodically monitors "show task replication" and "show database-replication summary" CLI command.

When replication error is found in two consecutive periodic, the script raise chassis alarm as follows

When the replication error state is cleared for two consecutive periodic , the script clears the alarm.

```

regress@R1_re> show chassis alarms

6 alarms currently active

Alarm time Class Description

2023-05-30 21:04:27 PDT Minor "database-replication state is not Synchronized"

2023-05-30 21:04:27 PDT Minor "Replication error for protocol LDP"

2023-05-30 21:04:27 PDT Minor "Replication error for protocol RSVP"

2023-05-30 21:04:27 PDT Minor "Replication error for protocol MPLS"

2023-05-30 21:04:27 PDT Minor "Replication error for protocol BGP"

2023-05-30 21:04:27 PDT Minor "Replication error for protocol OSPF"

```

## Syslog for alarm set events

```

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10007, color=YELLOW, class=CHASSIS, reason="Replication error for protocol OSPF"

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10010, color=YELLOW, class=CHASSIS, reason="Replication error for protocol BGP"

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10018, color=YELLOW, class=CHASSIS, reason="Replication error for protocol MPLS"

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10019, color=YELLOW, class=CHASSIS, reason="Replication error for protocol RSVP"

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10021, color=YELLOW, class=CHASSIS, reason="Replication error for protocol LDP"

R1_re alarmd[9939]: Alarm set: RE_DAEMON id=10099, color=YELLOW, class=CHASSIS, reason="database-replication state is not Synchronized"

```

## Syslog for alarm clear events

```

R1_re alarmd[9939]: Alarm cleared: RE_DAEMON id=10007, color=YELLOW, class=CHASSIS, reason="Replication error for protocol OSPF"

R1_re alarmd[9939]: Alarm cleared: RE_DAEMON id=10010, color=YELLOW, class=CHASSIS, reason="Replication error for protocol BGP"

R1_re alarmd[9939]: Alarm cleared: RE_DAEMON id=10019, color=YELLOW, class=CHASSIS, reason="Replication error for protocol RSVP"

R1_re alarmd[9939]: Alarm cleared: RE_DAEMON id=10021, color=YELLOW, class=CHASSIS, reason="Replication error for protocol LDP"

R1_re alarmd[9939]: Alarm cleared: RE_DAEMON id=10099, color=YELLOW, class=CHASSIS, reason="database-replication state is not Synchronized"

```

# Required configuration

```

set event-options generate-event 5MIN time-interval 300

set event-options policy task-replication-check events 5MIN

set event-options policy task-replication-check then event-script task-replication-check.slax

set event-options event-script file task-replication-check.slax

set event-options event-script optional

set system commit synchronize

```

# Author

[email protected]

# Script version

```

1.1 2025-05-01

```

*/

version 1.1;

ns junos = "http://xml.juniper.net/junos/*/junos";

ns xnm = "http://xml.juniper.net/xnm/1.1/xnm";

ns jcs = "http://xml.juniper.net/junos/commit-scripts/1.0";

ns str = "http://exslt.org/strings";

ns date = "http://exslt.org/dates-and-times";

import "../import/junos.xsl";

var $statefile="/tmp/task-replication-check.xml";

var $con = jcs:open();

var $alarmbase = 10000;

var $alarm_id_table := {

<ent> {

<id> 5;

<name>"OSPF2";

}

<ent> {

<id> 6;

<name>"OSPF3";

}

<ent> {

<id> 7;

<name>"OSPF";

}

<ent> {

<id> 8;

<name>"RIP";

}

<ent> {

<id> 9;

<name>"RIPng";

}

<ent> {

<id> 10;

<name>"BGP";

}

<ent> {

<id> 12;

<name>"ISIS";

}

<ent> {

<id> 13;

<name>"IGMP";

}

<ent> {

<id> 15;

<name>"DVMPR";

}

<ent> {

<id> 16;

<name>"PIM";

}

<ent> {

<id> 17;

<name>"MSDP";

}

<ent> {

<id> 18;

<name>"MPLS";

}

<ent> {

<id> 19;

<name>"RSVP";

}

<ent> {

<id> 20;

<name>"CCC";

}

<ent> {

<id> 21;

<name>"LDP";

}

<ent> {

<id> 23;

<name>"VPN";

}

<ent> {

<id> 24;

<name>"MVPN";

}

<ent> {

<id> 26;

<name>"L2VPN";

}

<ent> {

<id> 27;

<name>"L2CIRCUIT";

}

<ent> {

<id> 29;

<name>"VPLS";

}

<ent> {

<id> 30;

<name>"GMPLS";

}

<ent> {

<id> 31;

<name>"L-OSPF";

}

<ent> {

<id> 32;

<name>"SPRING-TE";

}

<ent> {

<id> 34;

<name>"L-ISIS";

}

<ent> {

<id> 35;

<name>"MLD";

}

<ent> {

<id> 43;

<name>"ESIS";

}

<ent> {

<id> 58;

<name>"EVPN";

}

template alarm($alarm-id,$action,$longalarm,$color="yellow",$curr-alarm) {

if ($action == "set") {

if (count($curr-alarm//alarm-detail[normalize-space(alarm-short-description) == $alarm-id]) == 0) {

var $rpc-set = {

<set-user-alarm> {

if ($color=="red") {

<red>;

} else {

<yellow>;

}

<alarm-id>$alarm-id;

<short-msg>$alarm-id;

<long-msg> $longalarm;

}

expr jcs:execute($con,$rpc-set);

}

} else if ($action == "clear") {

if (count($curr-alarm//alarm-detail[normalize-space(alarm-short-description) == $alarm-id]) > 0) {

var $rpc-clear = {

<clear-user-alarm> {

<id-number>$alarm-id;

}

expr jcs:execute($con,$rpc-clear);

}

match / {

<op-script-results> {

if ( $junos-context/re-master ) {

var $show-re = jcs:execute($con,<get-route-engine-information>);

if ($show-re//route-engine[mastership-state == "backup"]/status == "OK") {

var $curr-alarm = jcs:execute($con,<get-alarm-information>);

var $task-rep = jcs:execute($con,<get-routing-task-replication-state>);

var $database-sync = jcs:execute($con,<get-database-replication-summary-information>);

var $curr := {

<entries> {

<localtime>date:seconds();

for-each($task-rep/task-protocol-replication) {

<rep-state> {

var $dot = .;

<ent> {

<name>$dot/task-protocol-replication-name;

if ($dot/task-protocol-replication-state == "Complete") {

<state>"Complete";

} else {

<state>"Incomplete";

}

if ($database-sync//replication-database-type == "Synchronized") {

<database-sync> {

<state>"Synchronized";

}

} else {

<database-sync> {

<state>"Not Synchronized";

}

var $histfile = document($statefile);

if (($curr//localtime - $histfile//localtime < 600) && ($curr//localtime - $histfile//localtime > 0)) { /* ignoring too old histfile */

for-each ($histfile//rep-state/ent) {

var $ent = .;

if (($ent/state == "Complete") && ($curr//rep-state/ent[name == $ent/name]/state == "Complete")) {

call alarm($alarm-id=$alarmbase + $alarm_id_table/ent[name == $ent/name]/id,$action="clear",$curr-alarm=$curr-alarm);

} else if (($ent/state == "Incomplete") && ($curr//rep-state/ent[name == $ent/name]/state == "Incomplete")) {

call alarm($alarm-id=$alarmbase + $alarm_id_table/ent[name == $ent/name]/id,$action="set",$curr-alarm=$curr-alarm

,$longalarm="Replication error for protocol " _ $ent/name);

}

if (($histfile//database-sync/state == "Synchronized") && ($curr//database-sync/state == "Synchronized")) {

call alarm($alarm-id=$alarmbase + 99,$action="clear",$curr-alarm=$curr-alarm);

} else if (($histfile//database-sync/state == "Not Synchronized") && ($curr//database-sync/state == "Not Synchronized")) {

call alarm($alarm-id=$alarmbase + 99,$action="set",$curr-alarm=$curr-alarm

,$longalarm="database-replication state is not Synchronized");

}

<xsl:document method="xml" href=$statefile indent="yes"> {

copy-of $curr;

}

Modification History

2025-05-01 : Article Created