Integrating Splunk Observability Cloud alerts with ITSI
Splunk Observability Cloud alerts are what is known as stateful. That means an alert can be in different states, such as “ok” or “anomalous”. Any state change of an alert generates a new alert. After the original alert goes back to its normal state, an additional alert is created with a status of "ok", meaning everything is clear.
In Splunk Observability Cloud, the lifecycle of the stateful alert is called an incident. The incident number is carried with all alerts until the incident is cleared. The alert incident lifecycle should not be confused with an incident in Splunk On-Call incident response or ITIL/ITSM incident management. These are different.
You want to integrate Splunk Observability Cloud detector alerts into the ITSI Event Analytics workflow. To do this, Splunk Observability Cloud needs to send alerts from detectors to a Splunk Cloud Platform or Splunk Enterprise event index that Splunk ITSI is using.
This article is part of the Splunk Use Case Explorer for Observability, which is designed to help you identify and implement prescriptive use cases that drive incremental business value. It explains the solution using a fictitious example company, called CSCorp, that hosts a cloud native application called Online Boutique. In the AIOps lifecycle described in the Use Case Explorer, this article is part of Event analytics.
Solution
In this example, the Online Boutique application is being monitored by Splunk Observability Cloud detectors. When the detectors trigger, the alert needs to be sent to Splunk ITSI Event Analytics to be normalized, correlated, and grouped, and for notifications to be generated.
The goal is to set up and validate three different integration parts:
- Splunk HEC endpoint
- AWS Lambda function (review the Splunk Lambda function zip file and Splunk Lambda function yaml)
- Splunk Observability Cloud webhook
Watch this video to see how to configure the three parts of the integration.
Next steps
Still having trouble? Splunk has many resources available to help get you back on track.
- Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at OnDemand-Inquires@splunk.
com if you require assistance. - Splunk Answers: Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
- Splunk Customer Support: Contact Splunk to discuss your environment and receive customer support.
- Splunk Observability Training Courses: Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.