You might want to disk drive is nearing capacity when doing the following:
In order to execute this procedure in your environment, the following data, services, or apps are required:
Your organization's critical IT applications require space on disk to read and write to operate properly. When an application runs out of disk space, it usually results in application instability or crashing. You need a search to help detect when a disk drive is nearing capacity.
To optimize the search shown below, you should specify an index and a time range.
- In Splunk Enterprise or Splunk Cloud Platform, verify that you deployed the add-on to the search heads and Splunk Universal Forwarders on the monitored systems. For more information, see About installing Splunk add-ons.
- Run the following search:
| mstats avg(LogicalDisk.%_Free_Space) AS "win_storage_free" WHERE index="<name of your metrics index>" host="<name(s) of the host(s) you want to check>" instance="<name(s) of drive(s) you want to check>" instance!="_Total" by host, instance span=1m | eval storage_used_percent=round(100-win_storage_free,2) | eval host_dev=printf("%s:%s\\",host,instance) | timechart max(storage_used_percent) AS storage_used_percent BY host_dev
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
|| mstats avg(LogicalDisk.%_Free_Space) AS "win_storage_free" WHERE index="<name of your metrics index>" host="<name(s) of the host(s) you want to check>" instance="<name(s) of drive(s) you want to check)>" instance!="_Total" by host, instance span=1m||Search metrics index(es) where perfmon disk space data is being collected and filter down to the desired host(s) to check.|
|| eval storage_used_percent=round(100-win_storage_free,2)||Convert percent storage free to percent storage used for readability.|
|| eval host_dev=printf("%s:%s\\",host,instance)||Create a new field that combines the host and disk drive.|
|| timechart max(storage_used_percent) AS storage_used_percent BY host_dev||Plot the storage used for each host and disk over time.|
Create an alert based off this search so you can proactively manage potential stability issues.
- Ensure that you have the Splunk OTEL Collector installed on the host you want to monitor.
- In Splunk Infrastructure Monitoring, use the following SignalFlow to search the disk.utilization streaming metric, filter down to the desired host(s) and mountpoint(s), and summarize results by counting the total number of processes found per host.
A = data('disk.utilization', filter=filter('host', '<name of host to check>') and filter('mountpoint', '<name of disk to check>')).publish(label='A')
To alert when disk utilization is nearing capacity on the specified host(s) and mountpoint(s), use the SignalFlow from this procedure to configure a detector with an alert condition of "Resource Running Out" and alert settings of:
- Alert when nearing: Capacity
- Capacity: 100
- Trigger Sensitivity: Medium