Description

The event-options feature has a cache depth of 500 events. In certain scenarios, it does not get triggered when this cache fills up and needs to remove the earliest events it stored. 

This article explains how the event process cache works and what can be done to have it work with your configuration.

Symptoms

The event-options feature has a cache depth of 500 events. In certain scenarios, it does not get triggered when this cache fills up and needs to remove the earliest events it stored. One example where this cache may fill up is when RPM probes are used to trigger event-options. If there is a large number of RPM probes failing and the trigger values are high, the cache may not be able to hold enough instances of the RPM probe that needs to match the trigger values.

Solution

Example using RPM probes to trigger the event: 

[edit event-options]
policy CONNECTIVITY-PROBE-TEST {
    events ping_probe_failed;
    within 120 {
        trigger on 10;
        events ping_probe_failed;
    }
    attributes-match {
        "{$ping_probe_failed.test-name}" matches rpm_probe;
    }
    then {
        change-configuration {
            commands {
                "deactivate services";
            }
            commit-options {
                log "event-option occured";
            }
        }
    }
}

[edit services]
rpm {
    probe testprobe {
        test yvr_comcast_service {
            probe-type icmp-ping;
            target address 4.1.1.2;
            probe-count 15;
            probe-interval 4;
            test-interval 1;
            source-address 4.1.1.1;
            data-size 512;
        }
    }
}
With just this one RPM probe being configured on the MX and it is the interested event under the event-options having the trigger on 10 within 120 seconds is okay. Only the 10 failure events will be in the cache.

For example, if you add 79 more RPM targets to your MX that use the same intervals for their RPM configuration, the event-options configuration does not take into account whether these other targets fail or not. The event-options configuration will continue to only look at "{$ping_probe_failed.test-name} matches rpm_probe "

In a different scenario, lets say the core switch that the MX connects to reach these 80 RPM targets and have a catastrophic failure. At this time, every 5 seconds, there will be 80 ping failures from these 80 RPM probes. The event-options requires 10 failures for the specific RPM probe within 120 seconds. However, the issue now is the event queue will never be able to hold that many events, since these other 79 RPM targets are also in play.

All of these RPM probes will enter the event daemon cache and continue to remove the older entries in the cache too quickly since every 5 seconds, there will be 80 more events in the queue. This is how the event process cache works; it has to capture all of the RPM probe ping failures to find the matches it is looking for.

For this to work normally, set the event-options trigger to a value that can fit into the event daemons cache before it gets over written by newer events. A value of 5 would work in this last example. The RPM target in question can fail 5 times and have these events written into the event cache. Only a total of 400 new events would be written to the cache based on this configuration during the time frame it took for the RPM probe in question to fail 5 times.

Note that even though your event-options configuration is looking only at one RPM target, the event daemon still tracks each failure event for all of the RPM probes to determine if the RPM probe in question has failed enough times to fit the threshold you have set.