Skip to main content
Lantern Home
 
 
Splunk Lantern

Monitor CPU utilization for no-limit pod configuration situations

  1. Last updated
  2. Save as PDF

 

You work for a large organization that uses the Kubernetes container management platform. You don't have any pod limits set and are finding that more CPU usage can be consumed than intended, causing the resources of the node to run out.

Data required

Kubernetes data

How to use Splunk software for this use case

You can monitor node CPU usage within Splunk Infrastructure Monitoring to stay on top of this situation, reducing the impact to your customers.

  1. In Splunk Observability Cloud, click Alerts & Detectors in the left navigation pane.

    image1.png

  2. Click New Detector.

    image9.png

  3. Enter an appropriate name for the detector. In this example, K8s node - high cpu utilization is used. Then click Create Alert Rule.

    image6.png

  4. In the New Alert Rule, select Infrastructure or Custom Metrics Alert Rule and click Proceed to Alert Signal.

    image11.png

  5. Enter the metric name. The example below shows the metric cpu.utilization. Use the blue + icon to filter for k8s.node.name, then click Proceed to Alert Condition.

    image4.png

  6. Select Static Threshold to configure an alert when the node CPU has exceeded 90%. Click Proceed to Alert Settings.

    image5.png

  7. Enter the Alert settings for the conditions the alert will trigger on:
    1. Alert when: Select Above.
    2. Threshold: Set to 90.
    3. Trigger sensitivity: Select duration (to avoid any random spikes in CPU).
    4. Duration: Set to 5m.

      image3.png

  8. Click Proceed to Alert Message.
  9. In the Alert message:
    1. Set the alert Severity. You can choose from Critical, Major, Minor, Warning, or Info, depending on how you perceive the alert severity.
    2. (Optional) Enter a Runbook or dashboard URL, and enter a short tip for end users who might be troubleshooting the alert.
    3. (Optional) Customize the alert message further by clicking Customize.
    4. Click Proceed to Alert Recipients.

    image10.png

  10. Click Add Recipient to customize the recipients to send the alert message to. You can choose to add your own email address, any email, team, or webhook.

  11. Click Proceed to Alert Activation.
  12. Click Activate Alert Rule.

The appropriate recipients will now be notified when the alert rule conditions have been met.

Next steps

These additional Splunk resources might help you understand and implement these recommendations: