Skip to main content

 

Splunk Lantern

Maintaining *nix systems with Infrastructure Monitoring

 

In your organization, you have lots of *nix systems running critical applications or services. You need to monitor these systems to ensure the health of the associated apps and services, such as basic configuration, system diagnostics, file systems, and packages. You need to log and watch all these components, and ensure that appropriate technical staff are notified as quickly as possible if problems arise. With all these different concerns, you need Splunk searches that you can save and easily run on a schedule or as needed to keep your users up and running.

You can use Splunk software to manage patches and updates to ensure all connected systems and related processes are running after the patch or update is complete. You can also use Splunk software for a number of other maintenance tasks, such as watching out for connectivity issues.

Prerequisites

Technologies:

Data: 

How to use Splunk software for this use case

For each of the procedures below, ensure that you have the Splunk OTEL Collector installed on the host you want to monitor. 

*Nix CPU utilization nearing capacity

In Splunk Infrastructure Monitoring, use the following SignalFlow to search the cpu.utilization streaming metric and filter down to the hosts and processes you want to check. 

A = data('cpu.utilization', filter=filter('host.name', '<name of host to check>')).publish(label='A')

To alert when CPU utilization is nearing max capacity for the selected host(s) and process(es), use the SignalFlow from this procedure to configure a detector with an alert condition of "Static Threshold" and alert settings of:

  • Alert when: Above
  • Threshold: 95
  • Trigger sensitivity: Duration
  • Duration: 5m

*Nix memory utilization nearing capacity

In Splunk Infrastructure Monitoring, use the following SignalFlow to search the memory.utilization streaming metric and filter down to the desired host(s).

A = data('memory.utilization', filter=filter('host.name', '<name of host to check>'), rollup='latest').publish(label='A')

To alert when memory utilization is nearing max capacity for the selected host(s), use the SignalFlow from this procedure to configure a detector with an alert condition of "Static Threshold" and alert settings of:

  • Alert when: Above
  • Threshold: 95
  • Trigger sensitivity: Duration
  • Duration: 5m

Expected *Nix process not running

  1. Update the receivers section of the OTEL agent config file on the host to collect procstat metrics for each process.
    …
    receivers:
    …
      #The following config will collect process metrics for all processes. You can adjust the pattern parameter to filter down to a subset of processes
      smartagent/procstat:
        type: telegraf/procstat
        pattern: ".*"
  2. Update the services.pipelines.metrics.receivers section of the OTEL agent config file to include the procstat receiver.
    …
    service:
      extensions: …
      pipelines:
        traces: 
          …
        metrics:
          receivers: [..., smartagent/procstat]
          …
  3. In Splunk Infrastructure Monitoring, use the following SignalFlow to search the procstat.cpu_usage streaming metric, filter down to the desired hosts and processes, and summarize results by counting the total number of processes found per host.
    A = data('procstat.cpu_usage', filter=filter('host.name', '<name of host to check>') and filter('process_name', '<name of process to check>')).count(by=['host.name']).publish(label='A')

To alert when no process data is flowing in for the selected host(s) and process(es), use the SignalFlow from this procedure to configure a detector with an alert condition of "heartbeat" and alert settings of 15 minutes.

*Nix host stops reporting data

In Splunk Infrastructure Monitoring, use the following SignalFlow to search for hosts not reporting metrics after a fixed period of time.

A = data('cpu.utilization', filter=filter('host.name', '<HOSTS-TO-CHECK>')).publish(label='A')

The metric cpu.utilization is fundamental and should be present on all hosts. To alert when a host stops reporting data, use the SignalFlow from this procedure to configure a detector with an alert condition of "Heartbeat Check" and Alert Settings of "Hasn't reported For: 5m".

Next steps

To maximize their benefit, the procedures in this article likely need to tie into existing processes at your organization or become new standard processes. These processes commonly impact success with this use case: 

  • Running regular backups
  • Maintaining tooling for software provisioning 
  • Maintaining tooling for configuration management 
  • Site reliability engineering processes

Measuring impact and benefit is critical to assessing the value of IT operations. The following are example metrics that can be useful to monitor when implementing this use case:

  • Mean time to resolution 
  • Mean time to root cause
  • Reduction in defects

This use case is also included in the IT Essentials Learn app, which provides more information about how to implement the use case successfully in your IT maturity journey. In addition, these Splunk resources might help you understand and implement this use case:

Still need help with this use case? Most customers have OnDemand Services per their license support plan. Engage the ODS team at OnDemand-Inquires@splunk.com if you require assistance.