Skip to main content
Splunk Lantern

Integrating Splunk Observability Cloud alerts with ITSI

Splunk Observability Cloud alerts are what is known as stateful. That means an alert can be in different states, such as “ok” or “anomalous”. Any state change of an alert generates a new alert. After the original alert goes back to its normal state, an additional alert is created with a status of "ok", meaning everything is clear.

In Splunk Observability Cloud, the lifecycle of the stateful alert is called an incident. The incident number is carried with all alerts until the incident is cleared. The alert incident lifecycle should not be confused with an incident in Splunk On-Call incident response or ITIL/ITSM incident management. These are different.

You want to integrate Splunk Observability Cloud detector alerts into the ITSI Event Analytics workflow. To do this, Splunk Observability Cloud needs to send alerts from detectors to a Splunk Cloud Platform or Splunk Enterprise event index that Splunk ITSI is using.

This article is part of the Splunk Use Case Explorer for Observability, which is designed to help you identify and implement prescriptive use cases that drive incremental business value. It explains the solution using a fictitious example company, called CSCorp, that hosts a cloud native application called Online Boutique. In the AIOps lifecycle described in the Use Case Explorer, this article is part of Event analytics.

Solution

In this example, the Online Boutique application is being monitored by Splunk Observability Cloud detectors. When the detectors trigger, the alert needs to be sent to Splunk ITSI Event Analytics to be normalized, correlated, and grouped, and for notifications to be generated.

The goal is to set up and validate three different integration parts:

  1. Splunk HEC endpoint
  2. AWS Lambda function (review the Splunk Lambda function zip file and Splunk Lambda function yaml)
  3. Splunk Observability Cloud webhook

clipboard_e65ad44b7cd238f78fdba4e19450ecd54.png

Watch this video to see how to configure the three parts of the integration.

Next steps

Still having trouble? Splunk has many resources available to help get you back on track.