Skip to main content
Splunk Lantern

*Nix memory utilization nearing capacity

Excessive memory utilization on a host, particularly abnormal or prolonged, is a sign of potential issues with the critical applications running on the host. You want to detect when an application is starved for memory resources, so you can prevent performance degradations or application instability.

Procedure

Option 1

  1. Ensure that you have installed the Splunk Add-on for Unix and Linux on your Splunk search head, indexer, and the Splunk universal forwarders on the monitored systems. Click here for an example inputs.conf file that can be deployed to the universal forwarder on the *nix host to collect Memory utilization data and store the results into a metrics index.
  2. In Splunk Enterprise or Splunk Cloud Platform , run the following search. You can optimize it by specifying an index and adjusting the time range.
| mstats avg(vmstat_metric.memUsedPct) AS vmstat_metric.memUsedPct WHERE index="<name of *nix metrics index>" AND host="<name of host to check>" span=1m  BY host 
| timechart avg(vmstat_metric.memUsedPct) AS memUsedPct BY host

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation
| mstats avg(vmstat_metric.memUsedPct) AS vmstat_metric.memUsedPct WHERE index="< name of *nix metrics index >" AND host="<name of host to check>" span=1m  BY host  Search metrics index(es) where memory utilization data is being collected and filter down to the desired host(s) to check.
| timechart avg(vmstat_metric.memUsedPct) AS memUsedPct BY host Plot the percent of memory used for each host over time.

Next steps

Set up an alert based on this search so you can proactively manage potential stability issues. To alert when memory utilization is nearing max capacity, you can configure one of the following two recommendations:

  • Use the SPL from this procedure to configure a Core Splunk alert.
  • Configure the Average Free Memory vital metric for the Unix/Linux entity type in IT Essentials Work to alert when the Memory Utilization percentage is at or above a specific value.

Finally, you might be interested in other processes associated with the Maintaining *nix systems use case.

Option 2

  1. Ensure that you have the Splunk OTEL Collector installed on the host you want to monitor.
  2. In Splunk Infrastructure Monitoring, use the following SignalFlow to search the memory.utilization streaming metric and filter down to the desired host(s).
    A = data('memory.utilization', filter=filter('host.name', '<name of host to check>'), rollup='latest').publish(label='A')

Next steps

To alert when memory utilization is nearing max capacity for the selected host(s), use the SignalFlow from this procedure to configure a detector with an alert condition of "Static Threshold" and alert settings of:

  • Alert when: Above
  • Threshold: 95
  • Trigger sensitivity: Duration
  • Duration: 5m

Finally, you might be interested in other processes associated with the Maintaining *nix systems use case.