Description

In a QFX VC configuration log messages from a backup are not sent to the master member. This can be a problem in some cases for eg DMA errors.

If a backup member encounters a DMA error, the log messages are not sent to the master which in turn can be sent to an external Syslog server for proactive action

Solution

The following script can be deployed on BACKUP members only which will send selected messages (as per REGEX in the event script) to master

 

set event-options policy P1 events KERNEL
set event-options policy P1 attributes-match KERNEL.message matches "(.*DMA)|(.*ATA)"
set event-options policy P1 then event-script dma_script.slax
set event-options event-script traceoptions file escript.log
set event-options event-script traceoptions file size 1g
set event-options event-script traceoptions flag all
set event-options event-script file dma_script.slax
set event-options policy P1 then event-script dma_script.slax arguments message "{$$.message}"

On the master, please configure this
set system syslog file messages any any

set system syslog file messages structured-data

 

script location

 

/var/db/scripts/event

 

Script:-

version 1.1;
ns junos = "http://xml.juniper.net/junos/*/junos";
ns xnm = "http://xml.juniper.net/xnm/1.1/xnm";
ns jcs = "http://xml.juniper.net/junos/commit-scripts/1.0";
ns ext = "http://xmlsoft.org/XSLT/namespace";

import "../import/junos.xsl";
param $message;    
match / {
<event-script-result> {
 
   var $eid = event-script-input/trigger-event/id; 

   var $change_msg = {
         call escape-string($text = $message, $vec = '<>');
   }
   var $log = "logger fpc_backup " _ $change_msg;

   var $log1 = "logger fpc_backup " _ $message;
   expr jcs:trace($log);
   expr jcs:trace($log1);
   var $options := {
     <method> "junos-netconf";
     <username> "root";
     <passphrase> "Juniper";
    <routing-instance> "__juniper_private1__";
     }
    expr jcs:output("Testing jcs:empty jcs:open jcs:close jcs:output");
    var $conn = jcs:open("128.0.0.4", $options);
    if (jcs:empty($conn)) {
        expr jcs:output("open connection is empty ");
    } else {
        expr jcs:output("open connection  is not empty");
        var $rpc = {
            <request-shell-execute> {
                <command> $log;
            }
        }
        var $result = jcs:execute($conn,$rpc);
        copy-of $result;
    }
    expr jcs:close($conn);

}
}

template escape-string ($text, $vec) {

    if (jcs:empty($vec)) {
        expr $text;

    } else {
        var $index = 1;
        var $from = substring($vec, $index, 1);
        var $changed-value = {
            call replace-string($text, $from) {
                with $to = {
                    expr "\\";
                    expr $from;
                 }
            }
        }

        call escape-string($text = $changed-value, $vec = substring($vec, $index
 + 1));
    }
}
template replace-string ($text, $from, $to) {

    if (contains($text, $from)) {
        var $before = substring-before($text, $from);
        var $after = substring-after($text, $from);
        var $prefix = $before _  $to;

        expr $before;
        expr $to;
        call replace-string($text = $after, $from, $to);

    } else {
        expr $text;
    }
}

 

{master:0}

 

  1. Please be advised that the script may not work if the backup member crashes immediately after DMA errors are seen.
  2. Before triggering the logger command, please make sure the remote ip address is reachable and added to the known hosts list.
ssh [email protected] routing-instance __juniper_private1__
  1. In order to test the script, following highlighted command (or its variation) can be run on the backup-


{backup:0}[edit]
root@m03-34# run start shell
root@m03-34:BK:0% logger -e KERNEL this is ata error
root@t13-36:BK:1% logger -e KERNEL -a "message=/kernel ad1: FAILURE - WRITE_DMA status=41<READY,ERROR>error=4<ABORTED> LBA=7134840ad0: FAILURE - READ_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=500703"
root@m03-34:BK:0% exit

exit

{backup:0}[edit]
root@m03-34# exit
Exiting configuration mode

{backup:0}
root@m03-34> exit

root@m03-34:BK:0% exit
logout
rlogin: connection closed

{master:1}
root@m03-34> show log messages | last 10
May 20 06:07:20  m03-34 root: fpc_backup m03-34 logger: KERNEL: this is ata error

{master:0}

root@t13-36> show log messages|match dma
<13>1 2024-06-17T07:26:05.924-07:00 t13-36 root 11929 - - fpc_backup /kernel ad1: FAILURE - WRITE_DMA status=41<READY,ERROR>error=4<ABORTED> LBA=7134840ad0: FAILURE - READ_DMA status=41<READY,ERROR> error=4<ABORTED> LBA=500703


 
  1. All the messages sent from backup to master that match the REGEX in event script will have hostname as – ‘fpc_backup’


 
  1. Another way to trigger DMA error is to use the following command in shell . However, it is not recommended to use this command  as the system gets stuck in host shell and needs usb recovery


                    dd if=/dev/zero of=/dev/sda bs=1000M count=1

        dd if=/dev/zero of=/dev/sdb bs=1000M count=1



               DMA errors seen on master member

 

 

Modification History

2024-05-28 : Article Created