Description

Events are dropped in the Custom Rule Engine (CRE).

 

Symptoms

The following system notification or qradar.log/error entries are reported:

Sep 7 10:16:35 <ip address> [ecs] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=<hostname>:ecs0/EP/Processor2]] <domain>.cre.CRE: [WARN] [NOT:0060005100][<ip address>/- -] [-/- -]Custom Rule Engine has detected a total of 20487 dropped event(s). 20487 event(s) were dropped in the last 62 seconds. Queue is at 99 percent capacity

Solution

 

This situation indicates the Custom Rule Engine (CRE) has an expensive rule which is slowing down processing, and its queue has filled to the point where it is dropping events. In versions 2012 and above, the findExpensiceCustomRules script is located in /opt/qradar/support/

Note: You must run this command when the system is dropping events in the CRE. Running the tests after a reboot, ECS restart, or Hostcontext restart will not give an accurate measure of the expensive rules causing the drops.

The test will take some time to complete (4 to 5 minutes on a system under light load).

[root@<hostname>]# /opt/qradar/bin/findExpensiveCustomRules.sh
Writing custom rules output
Gathering data 100% /
Creating reports
Writing AverageExecutionTime
Writing TotalExecutionTime
Writing TotalExecutionCount
Writing MaximumExecutionTime
Writing TotalTestCount
Writing TotalTestTime
Writing AverageTestTime
Writing MaximumTestTime
Writing TotalActionsCount
Writing TotalActionsTime
Writing AverageActionsTime
Writing MaximumActionsTime
Writing TotalResponseCount
Writing TotalResponseTime
Writing AverageResponseTime
Writing MaximumResponseTime
Data can be found in ./CustomRule-<date>-<unique number>.tar.gz
Notes:
  • Use WinSCP to pull the file ./CustomRule-<date>-<unique number>.tar.gz down to your desktop.
  • Unpack the file with 7zip, WinRar, or equivalent.
  • In the results, you should find CustomRule-<date>-<unique number>.txt.
  • Open Excel and select CustomRule-<date>-<unique number>.txt with tab delimited.
  • You should now see all of the test values and rule names.
  • The Mbean column represents the rule names.
  • Total execution time and Average execution time are the key elements. You are looking for a high number (greater than 1) to see if you have an ineffective rule.
Also, it would be beneficial to get the contents of the whole /store/jheap directory from the Console.

Modification History

6-17-24 Updated product name and made more general.  Removed PII. Also updated Categories and visibility

Related Information