Skip to main content
Splunk Lantern

Building multi-KPI alerts in Splunk ITSI


​After you've set up and configured services within Splunk ITSI you can use the platform to look up the health of a service at any moment in time. But it's likely that you'll want to set up alerts as well so that you can be notified about changes to a service's KPIs, or when the overall Service Health Score (SHS) of any service degrades.

The recommended way to achieve scalable, enterprise-wide alerting for ITSI services and KPIs is by using the Content Pack for ITSI Monitoring & Alerting. You can see how to use the content pack in our product tip article, Using the Monitoring and Alerting Content Pack.

If you are unable to achieve your desired results using the Content Pack for Monitoring and Alerting, multi-KPI alerts may give you the additional control over the alerting conditions that you're looking for. This article explains how to set up multi-KPI alerts in Splunk ITSI, as well as configure actions associated with these alerts, such as sending an email to notify you when an alert is triggered.

Create a multi-KPI alert

  1. Open the Splunk ITSI app. By default, this will open on the Service Analyzer Tree View. If you're in Tile View, access the Tree View by clicking on the Tree button. image (7).png
  2. You can now see a hierarchy of all of your services. Click through the hierarchy to access the service you are looking for.
  3. Once you have clicked on the service, click Open all in Deep Dive to open the Deep Dive view of all its KPIs.
  4. At the top-left of the screen, click Bulk Actions, then click Create a Multi-KPI Alert.
  5. Select the KPIs you would like to build alerts from.
  6. You'll now be taken to the Multi-KPI Alerts screen. There are a few sections here you'll need to configure:
    1. Services. Here you can add or remove more services and their dependencies from your alert.
    2. KPIs in Selected Services. Here you can add KPIs from other services you selected in the first section to your alert.

    As you add services and KPIs to your alert, section 3 at the bottom of the screen, Selected KPIs, populates with the KPIs you have chosen.

    At the top-right of the screen, you can see Composite score displayed. This means that you'll be alerted if the composite health score of all services and KPIs in this alert fall below a threshold.

  7. Click Status over Time to configure the alert to notify you if the services and KPIs in this alert have been unhealthy for __ of the last __ minutes.
  8. Click Custom Time to configure the alert search timeframe.

  9. At the bottom of the screen, check the KPIs listed to make sure the ones you need are all included in the alert you want to build.

  10. For each of the KPIs listed in this area, click the link that reads # triggers set.
  11. Here, you can set the conditions that you want to trigger the alert. This works by taking into account the KPI's health score, the duration of that score and the timeframe you select. For example, if you set the search timeframe to 15 minutes, the health score to red (critical) and the duration to 50%, the alert will trigger when the KPI stays in critical for 50% of the last 15 minutes. Repeat this step for each of the KPIs you have selected.

  12. Click Save in the bottom-right corner of the screen once you have configured triggers for your KPIs.
  13. In the Create Correlation Search box, give your alert a name.
  14. Fill in Notable Event Title and Notable Event Description with the information you want to be shown when the alert triggers.
  15. Choose how often you want the alert to run.
  16. Choose a severity, then click Save.

    The last thing you'll probably want to do is to set up an action that occurs when this alert triggers - inclusion in an RSS feed, sending an email, or running a script.

  17. From the Correlation Searches screen, find the title of the alert you just configured. Click Edit, then By Multi-KPI Alerts Editor.

  18. Scroll down to Advanced Options, and expand it. Under Actions, configure the action you'd like to take place, and click Save.

Create a Service Health Score (SHS) alert

You might decide that you'd like to receive alerts when any service within ITSI becomes unhealthy, and you can configure SHS alerts in Splunk ITSI to do this.

  1. From the Service Analyzer top toolbar, click Configuration, then Correlation Searches.
  2. Find the search named Service Monitoring - Sustained Service Health Degradation (Recommended), and select the toggle to enable it.
  3. Click the Edit drop-down for this service, and click By Correlation Search Editor.
  4. You'll see that the fields in the Editor are pre-built to notify you when any service degrades. If you want to set up actions that occur when this alert triggers, scroll down to Advanced Options, and expand it. Under Actions, configure the action you'd like to take place, and click Save.

To see a record of triggered alerts, use the Episode Review menu option within the Service Analyzer toolbar. There, you can open episodes and view multiple related and grouped alerts.

Next steps

These additional Splunk resources might help you understand and implement these recommendations: