Sending Splunk Observability Cloud alerts to a Webex space

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

If you're already using Splunk Observability Cloud and Webex Messaging, you can bring alerts directly into the conversations you’re already having. It’s a simple way to improve visibility and reduce response times.

How to use Splunk software for this use case

Create a new Webex bot. A Webex bot lets you integrate services into Webex. This bot will receive alert notifications from the ThousandEyes webhook. For detailed instructions, see Creating a Webex Bot.
Copy the bot's access token. After setting up your new Webex bot, copy the bot's access token and save it somewhere easy to access. You'll need this token when you configure the Splunk Observability Cloud integration in Step 7.
(Optional) Create a space in Webex. In the Webex App, create a space to receive alert notifications from Splunk Observability Cloud. Skip this step if you already have a Webex App space where you'd like to receive alerts.
Add the Webex App bot to the space. In the Add people by name or email field, type the name name of the bot you want to add. In this example, the bot named "Splunk Observability Cloud Alerts" is added to the "Splunk O11y Cloud Example Alert Space" Webex App space. Then click Create.
Copy the space's roomId. You'll use the roomId with the bot's access token (from Step 2) to configure the ThousandEyes integration. To find the roomId:
1. Use your Webex App login to access the Webex API site.
2. Using a web browser, navigate to the following URL and click Run to look up the roomId: https://developer.webex.com/docs/api/v1/rooms/list-rooms.
Create a new Webhook integration in Splunk Observability Cloud.
1. Log into your Splunk Observability Cloud organization.
2. Select Data Management from the left navigation pane.
3. Click Add Integration.
4. Use the search box to locate the Webhook notification services.
5. Click the integration tile to start the setup.

Configure the webhook.

On the Webhook summary page, click Next.

Enter the following information into the Configure connection form, then click Next.

Field	Value
Name	Provide a descriptive name. This name will appear in the “Alert recipients” section of your detector configuration.
URL	https://webexapis.com/v1/messages
Method	POST
Shared Secret	Leave default
Headers
Key	Value
Content-type	application/json
Authorization	Bearer [token_from_step_2]

Keys and values might be case sensitive. In addition, in the Authorization header, the value should be in the following format: Bearer[space][token_value_from_step2]. For example if your token value from Step 2 was 123456789ABCDEFG, then the value of your Authorization header would be: Bearer 123456789ABCDEFG.

Copy and paste the payload template below into the Customize message payload. Update line 6 <YOUR_WEBEX_ROOM_ID> with the roomID obtained in Step 5. Remember to remove the angle brackets around your Webex room ID but leave the quotes. This template does the following, but you can customize it further to meet your team's needs.

Leverages Webex Adaptive Cards for a clean, structured alert view.
Includes a fallback to a standard text message if Adaptive Cards aren’t supported.
Uses Splunk Observability Cloud's custom webhook “if” logic to display optional fields like the detector’s Runbook URL and TIP only when they’re defined in the detector configuration.
Dynamically sets the message’s color using the severityDecoder helper function, aligning severity levels to appropriate visual cues. (See Splunk Webhook Helper Functions for details.)
Adds an actionable button that drives directly to Splunk Observability Cloud’s alert view - helping expedite troubleshooting right from the message.

When you're finished, click Next.

   {{!--
 Webex Room ID: Update this with the target Webex Room ID where notifications should be sent.
 Retrieve the Room ID using the Webex API or Webex Control Hub.
--}}
{
 "roomId": "<YOUR_WEBEX_ROOM_ID>",


 {{!--
   Markdown Fallback: Used if Adaptive Cards aren’t supported by the Webex client.
   Provides a readable plain-text version of the alert details. Best practice is to update this section as well as the adaptive card
 --}}
 "markdown": "<strong><u>{{{encodeString messageTitle}}}</u></strong>\n<strong>Severity:</strong> {{severity}}\n<strong>Status:</strong> {{{statusExtended}}}\n<strong>Source of Alert: </strong>{{{coalesce src 'src not found'}}}\n<strong>Alert Description:</strong>\n{{{messageBody}}}\n<strong>Detector: </strong>[{{{encodeString detector}}}]({{{detectorUrl}}}&orgID={{orgId}})\n{{#if runbookUrl}} [Runbook URL]({{runbookUrl}})\n{{/if}}{{#if tip}} <strong>Detector TIP specified:</strong>\n{{{encodeString tip}}}\n{{/if}}",


 "attachments": [
   {
     "contentType": "application/vnd.microsoft.card.adaptive",
     "content": {
       "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
       "type": "AdaptiveCard",
       "version": "1.3",


       "body": [
         {{!--
           Alert Title with Dynamic Severity Color.
           The severityDecoder helper function is used to determine the color based on severity.
           For more details, see Splunk Webhook Helper Functions:
           https://dev.splunk.com/observability/docs/integrations/webhook_integration_overview#Helper-functions
         --}}
         {
           "type": "TextBlock",
           "text": "{{{encodeString messageTitle}}}",
           "wrap": true,
           "weight": "bolder",
           "size": "large",
           "color": "{{{severityDecoder Critical='attention' Major='warning' Minor='accent' Warning='accent' Info='good' ok='good' default='default'}}}"
         },


         {{!-- Generic Alert Fields: Includes event timestamp, severity, status, source, and description --}}
         {
           "type": "TextBlock",
           "text": "Event Raised at: {{timestamp}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Severity: {{severity}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Status: {{{statusExtended}}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Source of Alert: {{{coalesce src 'src not found'}}}",
           "wrap": true
         },
         {
           "type": "TextBlock",
           "text": "Alert Description:\n{{{messageBody}}}",
           "wrap": true
         },


         {{!-- Optional TIP Section: Included only if 'tip' is present in the detector config--}}
         {{#if tip}}
         {
           "type": "TextBlock",
           "text": "Detector TIP specified:\n{{{encodeString tip}}}",
           "wrap": true
         },
         {{/if}}


         {{!-- Action Buttons: Provide quick access to investigate the alert or view the runbook (if one is defined in the detecor config) --}}
         {
           "type": "ActionSet",
           "actions": [
             {
               "type": "Action.OpenUrl",
               "title": "Investigate in Splunk Observability Cloud",
               "url": "{{{detectorUrl}}}&orgID={{orgId}}"
             }
             {{#if runbookUrl}},
             {
               "type": "Action.OpenUrl",
               "title": "Runbook",
               "url": "{{{runbookUrl}}}"
             }
             {{/if}}
           ]
         }
       ]
     }
   }
 ]
}

On the Review and save page, click Save.

Update Alert Receipts for your detectors. In the left navigation pane, select Detectors & SLOs, then select Detectors. Identify the detectors you’d like to receive Webex notifications about and select Manage subscriptions from the three dots menu.
In the Alert recipients dropdown menu, select Webhook and your newly-created webhook, then click Save.
When a detector alerts, you should see an Adaptive card appear in your newly created space.

Additional resources

The content in this guide comes from a Splunk Community Blog, How to Send Splunk Observability Alerts to Webex teams in Minutes, one of the thousands of Splunk resources available to help users succeed. You might find additional valuable content on our Cisco data descriptor page.