Configuring ITSI correlation searches for monitoring episodes
You have Splunk ITSI episodes being created in ITSI from Splunk Observability Cloud alerts, so now you want to create two episode monitoring correlation searches.
- The first controls when a Splunk On-Call incident should be created.
- The second determines if the episode’s severity should be elevated or degraded due to incoming notable events into the episode, and if the episode should be closed due to self-healing criteria.
This article is part of the Splunk Use Case Explorer for Observability, which is designed to help you identify and implement prescriptive use cases that drive incremental business value. It explains the solution using a fictitious example company, called CSCorp, that hosts a cloud native application called Online Boutique. In the AIOps lifecycle described in the Use Case Explorer, this article is part of Event analytics.
Solution
Use the Content Pack for ITSI Monitoring and Alerting monitoring correlation searches. The Content Pack provides many examples of these searches, but this article will explore two critical ones to start with so you can see quick value. As your implementation grows then you can use additional monitoring correlation searches, or even create custom ones, to help you solve your more complicated use cases. Before you can create these searches, ensure you have completed the following steps:
- Integrated Observability Cloud alerts with Splunk ITSI
- Normalized Observability Cloud alerts into the ITSI Universal Alerting schema
- Configured ITSI correlation searches to create notable events
- Configured action rules in the ITSI Notable Event Aggregation Policy for Splunk On-Call Integration
These two episode monitoring correlation searches evaluate all open episodes and create new notable events when a new Splunk On-Call incident needs to be created or when an episode state change occurs. These new notable events become part of the associated episode.
Next, the ITSI rules engine, which runs the NEAP Policy, applies action rules against the newly created notable events. If the action rule's specific activation criteria matches against the notable event data, then an action (such as creating a Splunk On-Call incident) is performed as defined in the action rule.
This design pattern is an integral part of the ITSI Monitoring and Alerting content pack and is explained further in the following video.
Watch this video to see how to configure and deploy these two ITSI episode monitoring correlation searches, as well as validate the creation of the notable events and the action rule processing.
You can download this ITSI Backup file that includes three correlation searches and one Notable Event Aggregation Policy (NEAP). Use the ITSI Backup/Restore utility to restore these artifacts into your instance of Splunk ITSI.
Next steps
Still having trouble? Splunk has many resources available to help get you back on track.
- Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at OnDemand-Inquires@splunk.
com if you require assistance. - Splunk Answers: Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
- Splunk Customer Support: Contact Splunk to discuss your environment and receive customer support.
- Splunk Observability Training Courses: Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.