You might need to be alerted to high CPU utilization when doing the following:
In order to execute this procedure in your environment, the following data, services, or apps are required:
- Splunk Enterprise or Splunk Cloud Platform
- System log data
High CPU utilization can be an indication that the host is having problems. If your system is overutilized, it does not have enough capacity for the CPU demand. You want to use metrics to detect heavy CPU usage before it impacts system performance, then alert when that metric exceeds a specified threshold.
To optimize the search shown below, you should specify an index and a time range.
- Run the following search:
| mstats avg(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND index=<metrics index name> AND instance="_Total" span=1m BY host | stats avg(_value) AS cpu_usage BY host | eval Critical_Usage = if(cpu_usage > 95, "Yes", "No") | table host Critical_Usage cpu_usage | where Critical_Usage="Yes"
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
|| mstats avg(_value) prestats=true WHERE metric_name="Processor.%_Processor_Time" AND index=<metrics index name> AND instance="_Total" span=1m BY host||
Use the %_Processor_Time values in a metrics index to determine average CPU utilization, setting prestats to true so the data can be used with the stats command.
If you don't the names of your available metrics, run this search first separately: | mcatalog values(metric_name) WHERE index=*
|| stats avg(_value) AS cpu_usage BY host||
Display the average CPU utilization for each host in a cpu_usage field.
|| eval Critical_Usage = if(cpu_usage > 95, "Yes", "No")||Set the field named Critical_Usage to show whether CPU usage has exceeded 95 percent.|
|| table host Critical_Usage cpu_usage||Display the results in a table with columns in the order shown.|
|| where Critical_Usage="Yes"||Filter the results to show only those that have passed the Critical Usage threshold.|
You can modify this search to alert on disk utilization and memory utilization. You can also adjust the threshold to meet your organizational needs. After you configure the search, save it as an alert and customize the trigger actions.