This article is written to describe how to call a shell script when a specific log message is encountered on the device This can be helpful in collecting logs during intermittent issues (particularly when the issue span time is less and manual collection of logs is not possible) and identify the root cause for the issue. 

The script can be used and modified according to your own requirements. Customer was recording the below log message and was subsequently seeing drops on the PFE.

May 1 02:26:51 lab-qfx5100 : %PFE-3: fpc0 Maximum number of ECMP entries reached, nh_idx:131093

However these drops cannot be captured in any log messages and the time for which the drops are seen is very short. Customer needs an event script where the system can monitor for the log stamp and collect the required command outputs automatically at first detection.

1. Open Shell prompt and get under /var/tmp/folder

start shell user root

cd /var/tmp/folder 

2. Create a file under this folder

cat > execute.sh

3. Copy the below contents. Replace the "Target Message" and command for which the outputs need to be collected.

#!/bin/sh

# Log file to monitor (replace with actual log file path)

LOG_FILE="/var/log/messages"

# Specific log message to watch for

TARGET_MESSAGE="Maximum number of ECMP"

# Function to collect output from PFE Broadcom shell

collect_output() {

 # Use 'cprod' command to collect output from PFE shell. Similarly you can use 'cli -c' to run cli commands.

 cprod -A fpc0 -c 'set dcbcm bcm "show c"' >> /var/tmp/outage_data.txt

 cprod -A fpc0 -c 'set dcbcm bcm "show c cpu"' >> /var/tmp/outage_data.txt

 cli -c "show interfaces terse" >> /var/tmp/outage_data.txt

}

# Monitor the log file continuously

tail -n0 -F "$LOG_FILE" | while read line; do

 # Check if the line contains the target message

 if echo "$line" | grep -q "$TARGET_MESSAGE"; then

  echo "Detected target message: $line"

  # Collect output from PFE Broadcom shell when target message is detected. The number of times you call the function collect_pfe_output(), the script will collect the output for each function call.

  echo "Collecting output from PFE Broadcom shell:"

  collect_output

  collect_output

  collect_output

 fi

done

4. To Run the script.

 nohup sh /var/tmp/execute.sh &

nohup will ensure that the script is not terminated even if the SSH session is disconnected.