Reducing skipped searches

User or role quota limit has been reached

If you have programmed user or role quotas, certain searches may skip if these quota limits are breached. This is the intended function of quotas - to limit the number of concurrent searches a user or users within a role can run concurrently. You can change user or role quotas to fix this.

Sometimes you will also see that searches from certain users are skipped because the user doesn't have permissions to run scheduled jobs. Here is a good resource on resolving this issue.

The maximum number of concurrent running jobs for a historical scheduled search has been reached

Scheduled searches run at a certain frequency (e.g. every 5 minutes) and by default only 1 instance of a scheduled search can run at any given time. This limit is defined by <max_concurrent> in the Splunk Enterprise savedsearches.conf file and generally, you don’t need to change this attribute to more than 1 (default). If a scheduled search job cannot be completed before the next period starts, it will be skipped. For instance, if a job takes 10 minutes to complete and runs at a 5 minute frequency, then it will skip.

To fix this problem, you need to identify which scheduled searches are skipped due to this reason and then do one of the following:

• Change the schedule frequency so it is higher than execution time for the search.
• Or, try to reduce the search execution time:
  • If you are resource constrained, you can use workload management to put these searches in a high priority pool to get them more resources so they complete faster.
  • Check the load on indexers to see if they are resource constrained and you need to increase resources. You can run this search or look at the monitoring console to see CPU utilization on indexers:

    index=_introspection host=idx* component=Hostwide
    | eval total_cpu_usage = 'data.cpu_system_pct' + 'data.cpu_user_pct'
    | timechart span=10m perc95(total_cpu_usage) BY host
    

    If the output looks like the image below, this is why searches have a long execution time. In this instance, you'll need to add more indexer resources.

    

The maximum number of concurrent auto-summarization searches has been reached

The number of summarization searches that can run concurrently is capped and that limit is defined by <auto_summary_perc> in the limits.conf file. This attribute may be raised to 75% to allow auto-summarization searches to be a higher percentage of the overall scheduled search limit, reducing the skipped searches.

You can also check if you are seeing skipped searches at a particular time, such as the top of the hour or 30 minutes past the hour. This means you have scheduled too many searches at the same time. You can resolve this issue by allowing the scheduler to optimize the search dispatch time by setting <allow_skew> or <schedule_window>.

The maximum number of concurrent historical scheduled searches on an instance or cluster has been reached

This is the most common reason for skipped searches. This happens because the searches have taken all available search slots so new searches cannot be scheduled.

index=_internal sourcetype=scheduler savedsearch_name=* status=skipped 
| bin _time=1h 
| stats count by _time