In this step, alerts are normalized, correlated, and grouped into meaningful actionable groups called episodes in Splunk ITSI. Episodes can be of one of two types, monitoring or observational.
- Monitoring episodes are system status changes that need a break/fix process initiated. An example is when an EC2 CPU utilization exceeds a threshold and needs remediation.
- Observational episodes are where something was observed and probable or root cause needs to be determined. An example is a sudden spike in error rate for an application operation endpoint that has occurred out of normal.
The guides in this section and in the Notification focal area use a fictitious company called CSCorp that hosts a cloud native application called Online Boutique deployed in a Kubernetes cluster. The guides show how the different focal areas in the Full Observability Stack can be consolidated. In the context of the application Online Boutique, the alerts coming from Splunk APM, Splunk Real User Monitoring, and Splunk Synthetic Monitoring detectors are consolidated as a single observational episode. The guides also demonstrate monitoring the Kubernetes cluster and sub-components for alerts. The pink arrows in the diagram above, (ITSI - Event Analytics) points to the area we will be focusing on. Also, note the pink dotted lines, which represent an AIOps workflow that harvests the insights from the observe stage (metrics, traces, logs, and alerts), correlates and notifies (the Engage stage) as an Splunk On-Call incident in the context of a service or application (the Act stage).
Before you begin using the prescriptive outcome guides, use the Splunk Docs for the Content Pack for ITSI Monitoring and Alerting and the following diagram to familiarize yourself with the flow of an alert from Splunk Observability Cloud as comes into Splunk ITSI and gets processed, grouped (episode), and acted up (alert action). Go through the diagram a few times so it becomes second nature as you expand your observability capabilities.
Note that the examples linked below treat Splunk Observability Cloud detector as an external alerts source. Also, the content pack is a framework, so the guidance makes minor adjustments to correlation searches and Notable Events Aggregation Policies (NEAPs) for this full end-to-end solution. You will do the same when you deploy it.
Event analytics prescriptive outcome guidance
Each of these guides build upon each other, so we recommend you do these in sequential order if this is your first time reviewing them. You will apply the concepts taught to your environment, so you will need your own licenses or trial versions of the Splunk products. Click the Free Splunk button in the upper-right corner if you do not have an environment. In addition, we have provided this ITSI backup file that includes three correlation searches and one Notable Event Aggregation Policy (NEAP). Use the ITSI Backup/Restore utility to restore these artifacts into your instance of Splunk ITSI.
- Integrating Observability Cloud alerts with Splunk ITSI
- Normalizing Observability Cloud alerts into the ITSI Universal Alerting schema
- Configuring ITSI correlation searches to create notable events
- Configuring the ITSI Notable Event Aggregation Policy
When you have reviewed these five guides, move on to Notification.