Skip to main content
Splunk Lantern

Introduction to working with alerts in Splunk InfraMon

To help maximize the value of your data, the friendly Splunk Customer Success team created this quick reference list that highlights how customers can best start configuring alerts in Splunk Infrastructure Monitoring. 

  • Create detectors. Detectors are the configurable resources in-app that monitor metrics on a plot line, trigger alert events, and clear events based on conditions you define in rules. You have a number of starting points when creating a detector -- you can: clone an existing detector, create a detector from a chart, or create from the API. An important concept with detectors is that you are essentially creating a chart for the analytics engine to analyze and monitor, so keep that in mind when creating these. If you want to alert off an important chart you just created, create the detector from that chart. You can also create a detector from scratch in the UI. When doing so, you must establish:
    • Detector rules.
      • When the detector will be triggered, based on conditions related to the detector’s signal/metric
      • The severity of the alert to be generated by the detector
      • Where notifications should be sent
      • Type. Choose what type of detector to create: APM Metric or Infrastructure/Custom Metric.
      • Alert Signal. Decide what metric are you trying to alert on and apply filters, analytics, and formulas.
      • Alert Condition. Define the conditions of the signal/metric in which you would like to be alerted on. A straightforward example is ‘Static Threshold’  or ‘Heartbeat check’ and a more complex example is ‘Custom Threshold’ where you can compound conditions using AND or OR logic.
      • Alert Settings. These settings will depend on which condition is selected and will be configured at this step.
      • Alert Message. Define the severity of the alert and customize the message of it. Can also link to helpful documentation to be delivered with the alert.
      • Alert Recipients. Define who will receive the alert and the delivery method, such as email, Splunk On-Call, Slack, PagerDuty, or Webhook.
  • View alerts. The Alerts page gives you a holistic view into active alerts. You can also filter alerts to zero-in on the most critical active issues. Click any item in the list to view details about the alert. In the details popup, you can click Resolve to set the alert’s status to “Resolved,” click View in detector to open the detector that triggered the alert (see Viewing active alerts for a specified detector), or click Close to return to the alerts list.
  • Filter alerts. You can click on any of the five large alert counters to drill down into alerts of that single severity level; a filter for severity level is added. You can also use the Filter field to show only alerts that are relevant to particular tags or dimensions.
  • Set up notifications. To get the most out of the real-time streaming nature of Splunk Infrastructure Monitoring you’ll likely want to integrate it with a another service for means of notification, like Splunk On-Call, PagerDuty, or Slack. Doing so will help you respond more efficiently which ultimately compliments the, again, real-time streaming nature.