Skip to main content
Lantern Home
 
 
Splunk Lantern

Preventing concurrency issues and skipped searches

  1. Last updated
  2. Save as PDF

 

A correlation search scans multiple data sources for defined patterns. When the search finds a pattern, it performs an adaptive response action. This is a powerful capability, but when multiple correlation searches all run at the same time, you can have search concurrency issues and skipped searches.

Solution

The information in this article applies to Splunk Enterprise Security (ES) versions 7.x. If you have upgraded to Splunk Enterprise Security version 8.x, some terminology and steps might not apply. For additional assistance on this use case with ES 8.x, Splunk Professional Services can help.

Identify overcrowded scheduling and then shuffle the searches around to ensure they aren't all running at the same time.

  1. (Optional) Run the following SPL to list out the cron schedules used by the enabled searches. The results give you a high-level overview of overcrowded schedules so you can easily identify problem areas. You might want to create a pie chart visualization of the results and add it to a dashboard for monitoring. 
    | rest splunk_server=local /servicesNS/-/-/saved/searches search="is_scheduled=1" search="action.correlationsearch.enabled=1" search="disabled=0"
| fields title author eai:acl.app eai:acl.sharing cron_schedule dispatch.earliest_time dispatch.latest_time
| stats count AS CronCount BY cron_schedule
| sort - CronCount
  2. Run the previous search again, but change the stats command to eventstats to list out details for each of the enabled searches.
    clipboard_e279de2a6afc1df506cfa9c4e0aad68e1.png
  3. Make note of searches with the same cron schedule.
  4. To edit these searches, go to Configure > Content > Content Management.
  5. Click the name of a search to open the editing page.
    clipboard_ee11c9346d72bff485723768af79544ac.png
  6. In the Time Range section, manually adjust the Cron Schedule. For example:
    • If the alerts are run hourly, then stagger which minute the alert starts on. Set your first search to 1 * * * *. Then open another search and set it to 2 * * * *. Then 3* * * * for a third search and so on.
    • If the searches need to run every 5 or 10 minutes, you can splay the cron schedules to allocate searches to run on different minutes.
      • For five-minute intervals, this would be 0-55/5 * * * * for the first search, 1-56/5 * * * * for the second search, then 2-57/5 * * * *, 3-58/5 * * * *, 4-59/5 * * * *, and so on.
      • For ten-minute intervals, this would be 0-50/10 * * * * for the first search, 1-51/10 * * * * for the second search, then 2-52/10 * * * * and so on.
  7. In the Time Range section, for the search window, you should account for a delay in ingestion and data model acceleration. We recommend the following settings:
    • Earliest Time:-70m@m
    • Latest Time: -10m@m

Next steps

These additional Splunk resources might help you understand and implement this product tip: