Building multi-KPI alerts in Splunk ITSI

Create a multi-KPI alert

1. Open the Splunk ITSI app. By default, this will open on the Service Analyzer Tree View. If you're in Tile View, access the Tree View by clicking on the Tree button. 
2. You can now see a hierarchy of all of your services. Click through the hierarchy to access the service you are looking for.
3. Once you have clicked on the service, click Open all in Deep Dive to open the Deep Dive view of all its KPIs.
4. At the top-left of the screen, click Bulk Actions, then click Create a Multi-KPI Alert.
5. Select the KPIs you would like to build alerts from.
6. You'll now be taken to the Multi-KPI Alerts screen. There are a few sections here you'll need to configure:
   1. Services. Here you can add or remove more services and their dependencies from your alert.
   2. KPIs in Selected Services. Here you can add KPIs from other services you selected in the first section to your alert.

   As you add services and KPIs to your alert, section 3 at the bottom of the screen, Selected KPIs, populates with the KPIs you have chosen.

   

   At the top-right of the screen, you can see Composite score displayed. This means that you'll be alerted if the composite health score of all services and KPIs in this alert fall below a threshold.

7. Click Status over Time to configure the alert to notify you if the services and KPIs in this alert have been unhealthy for __ of the last __ minutes.
8. Click Custom Time to configure the alert search timeframe.

   

9. At the bottom of the screen, check the KPIs listed to make sure the ones you need are all included in the alert you want to build.

   

10. For each of the KPIs listed in this area, click the link that reads # triggers set.
11. Here, you can set the conditions that you want to trigger the alert. This works by taking into account the KPI's health score, the duration of that score and the timeframe you select. For example, if you set the search timeframe to 15 minutes, the health score to red (critical) and the duration to 50%, the alert will trigger when the KPI stays in critical for 50% of the last 15 minutes. Repeat this step for each of the KPIs you have selected.

    

12. Click Save in the bottom-right corner of the screen once you have configured triggers for your KPIs.
13. In the Create Correlation Search box, give your alert a name.
14. Fill in Notable Event Title and Notable Event Description with the information you want to be shown when the alert triggers.
15. Choose how often you want the alert to run.
16. Choose a severity, then click Save.

    

    The last thing you'll probably want to do is to set up an action that occurs when this alert triggers - inclusion in an RSS feed, sending an email, or running a script.

17. From the Correlation Searches screen, find the title of the alert you just configured. Click Edit, then By Multi-KPI Alerts Editor.

    

18. Scroll down to Advanced Options, and expand it. Under Actions, configure the action you'd like to take place, and click Save.

    

Create a Service Health Score (SHS) alert

You might decide that you'd like to receive alerts when any service within ITSI becomes unhealthy, and you can configure SHS alerts in Splunk ITSI to do this.

1. From the Service Analyzer top toolbar, click Configuration, then Correlation Searches.
2. Find the search named Service Monitoring - Sustained Service Health Degradation (Recommended), and select the toggle to enable it.
3. Click the Edit drop-down for this service, and click By Correlation Search Editor.
4. You'll see that the fields in the Editor are pre-built to notify you when any service degrades. If you want to set up actions that occur when this alert triggers, scroll down to Advanced Options, and expand it. Under Actions, configure the action you'd like to take place, and click Save.

   

Next steps

These additional Splunk resources might help you understand and implement these recommendations:

• Splunk Docs: Overview of the Service Analyzer in ITSI
• Blog: A blueprint for ITSI alerting