Skip to main content
 
 
Splunk Lantern

Integrating Splunk Observability Cloud alerts with Cloud Platform or Enterprise

 

In order to start implementing the workflow described in Managing the lifecycle of an alert: from detection to remediation, you need to first get Splunk Observability Cloud sending alerts from detectors to a Splunk Cloud Platform or Splunk Enterprise event index that Splunk ITSI is using, making them available for Splunk ITSI's use.

Solution

The diagram below shows the overarching architecture for the integration.

The scope for this article is surrounded by the pink box in the diagram. The boxed area shows that Splunk Observability Cloud alerts are sent to Splunk Cloud Platform or Splunk Enterprise via a webhook, configured as a recipient in the Splunk Observability Cloud detectors. The Splunk Observability Cloud alert events are processed through a Splunk Cloud Platform or Splunk Enterprise HTTP Event Collector (HEC) endpoint, and stored in a custom index.

The protocol you use in this process must be https with an appropriate port such as 443 or 8443. It must be secured by a trusted certificate authority for the certificate. Splunk Observability Cloud does not support self-signed certificates. These requirements are enforced by the Splunk Observability Cloud webhook configuration.

About alerts and incidents in Splunk Observability Cloud

Splunk Observability Cloud alerts are what is known as stateful. That means an alert can be in different states, such as “ok” or “anomalous”. Any state change of an alert generates a new alert. After the original alert goes back to its normal state, an additional alert is created with a status of "ok", meaning everything is clear.

In Splunk Observability Cloud, the lifecycle of the stateful alert is called an incident. The incident number is carried with all alerts until the incident is cleared. The alert incident lifecycle should not be confused with an incident in Splunk On-Call incident response or ITIL/ITSM incident management, as these are different.

Procedure

The overall procedure consists of a number of different steps:

  1. Set up the event index.
  2. Set up the HTTP Event Collector (HEC) endpoint.
  3. Set up the Splunk Observability Cloud webhook.
  4. Check that everythiing works as intended.

1. Set up the event index

Create an events-based index. Consult the Docs for detailed instructions on how to do this in Splunk Cloud Platform and Splunk Enterprise.

You should follow appropriate naming standards for the index name. This example uses the name "alerts".

2. Set up the HTTP Event Collector (HEC) endpoint

Consult the Docs for detailed instructions on how to do this in Splunk Cloud Platform and Splunk Enterprise.

  1. Navigate to Settings > Data Inputs > HTTP Event Collector.

  2. In Select Source, create a new token named "o11y_alerts_webhook" with a source name of "o11y_alerts_webhook", then click Next.

  3. In Input Settings, leave the Source type field as it is. Add “alerts” to the list of allowed indexes, change the default index to “alerts”, then click Review.

  4. Verify the settings are correct and click Submit. Otherwise, click < to make changes.

  5. Test the HEC endpoint.

    1. On Splunk Cloud Platform, run the following on the command line:
      curl https://http-inputs-[stack].splunkcloud.com:443/services/collector/raw -H "Authorization: Splunk XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -d '{"event": "Testing HEC token for o11y_alerts_webhook"}'
      A successful response is indicated with {"text":"Success","code":0}.

    2. On Splunk Enterprise, run the following curl command on the command line: curl https://http-inputs-[stack].splunkcloud.com:443/services/collector/raw -H "Authorization: Splunk XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -d '{"event": "Testing HEC token for for o11y_alerts_webhook"}'A successful response is indicated with{"text":"Success","code":0}

  6. Run the following search to validate that the data has been indexed.

source="o11y_alerts_webhook" (index="alerts")

3. Set up the Splunk Observability Cloud webhook

  1. Click Data Management.

    clipboard_e32c604a32513f2cbf6376f1e6cb2da48.png

  2. Click the Add Integration button. 

    image10.png

  3. Enter "Webhook" in the search box and click on the search result.

    image16.png

  4. Click the New Integration button to add an integration instance to support the webhook for sending an Splunk Observability Cloud alert to the Splunk HEC endpoint.

    image3.png

  5. Enter the following. See Splunk Docs for full configuration details.
    1. Name: Enter a name, using an appropriate naming convention.
    2. URL: The URL to the Splunk Cloud Platform or Splunk Enterprise HEC endpoint. This must use HTTPS secured protocol as well as a known HTTPS port such as 443 or 8443. For example, enter https://<HEC endpoint address>:<https port>/services/collector/raw
    3. Shared Secret: This is not required, so leave it blank.
    4. Headers: Click the Add Header button. In Key, enter "Authorization". In Value, add the HEC token value from step 1. You need to include the prefix of "Splunk".image7.png

 

Your final configuration should look like this:

image4.png

  1. Validate the creation of the webhook by clicking the ... next to your newly-created webhook instance, then click Validate. You should receive confirmation of the validation.image8.png

4. Check that everything works as intended

  1. Set up a test detector in Splunk Observability Cloud.
    1. Add the webhook as the recipient.
    2. Create a detector threshold to simulate an alert. Set the threshold on the metric with a static value to control when it is alert.
  2. Check that the alert has been triggered for the detector by navigating to a triggered alert and ensuring it shows the new webhook as the recipient under "About this Alert".image15.png
  3. Check that the triggered Splunk Observability Cloud alert has been sent into the alert index in Splunk Cloud Platform or Splunk Enterprise by running the following search: source="o11y_alerts_webhook" (index="alerts")

Next steps

Now that your alerts from an Splunk Observability Cloud detector are being sent to the Splunk Cloud Platform or Splunk Enterprise HEC endpoint for Splunk ITSI processing, the next step you should perform is Normalizing Observability Cloud alerts into the ITSI Universal Alerting schema.

Still having trouble? Splunk has many resources available to help get you back on track.