Skip to main content
Lantern Home
 
 
Splunk Lantern

Configuring the Splunk On-Call integration with IT Service Intelligence

  1. Last updated
  2. Save as PDF

 

Your company uses Splunk ITSI to normalize, correlate, and group related alerts, as well as generate incident notifications. Now, you want these critical notifications sent to Splunk On-Call to quickly get actionable Splunk ITSI episodes to the right teams so they can resolve problems.

This article is part of an overall alert management workflow that's described in Managing the lifecycle of an alert: from detection to remediation. To implement the entire workflow, before you follow these steps, make sure you have done the following:

Solution

  1. Enable the integration for Splunk ITSI from Splunk On-Call. To enable this integration, from Splunk On-Call’s web portal, navigate to Integrations > 3rd Party Integrations > Splunk ITSI > Enable Integration. This generates a Service API Endpoint and API Key. Copy the API Key to the clipboard as you'll need it in a later step. The screenshot below shows where the API key can be seen, although in this example it's blacked out for privacy reasons. 

    clipboard_ea1c413cdfc9629c0291d638a139d363d.png

  2. Download the Splunk On-Call (VictorOps) app from Splunkbase and install it on Splunk Enterprise or {{cloud)). For more information on how to install any Splunk-supported add-on to your Splunk platform, refer to Splunk Docs. You must restart {{enterprise}} to complete the install. After you install it, the app can be seen under Apps in the left-hand navigation pane.

    clipboard_eec20cc0cbd98492062373877b8edcf0a.png

  3. Open the app to begin the configuration from the Configure Splunk On-Call home page.

    image4.png

  4. Click Setup Account. On this page:
    1. Specify the organization slug or organization name associated with the Splunk On-Call account. If you're not sure what your organization slug is for your account, you can find it from the URL you use to access Splunk On-Call, for example, https://portal.victorops.com/client/org-slug.
    2. Click the Submit button, and you should get the message “Successfully Saved Organization Slug”.
    3. Click the Dismiss button.
    4. Click Setup Alert API Key to move to the next step. 

      clipboard_e1b6081910c7b7515ee58623858e0f7c0.png

      clipboard_e578dbb6b88958975ee8e76da2b579c4c.png

  5. On the Alert API Key Configuration page, the Name (org-slug) field is pre-populated for you. Complete the following:
    1. Enter the API key that you copied to your clipboard in step 1, and also enter the Routing Key. The routing key controls the routing of incidents to specific teams. If the routing key is left blank, alerts will be routed to your default routing key.
    2. Click the Add Key button. You should see your new configuration added to the list. 
    3. Test your configuration by selecting the Send Test Alert link under Actions. This should return a “Successfully sent test alert!” message. 
    4. Click Dismiss.
    5. Click Setup Data API Key to proceed to the next step. 

      image4.jpg

      image11.png

      This test alert will not be marked as an incident in your org, as it is an info alert. To find this test alert, you’ll need to look in your Timeline instead of the Incidents tab. If you want to have the alert marked as an incident, you can go to Search & Reporting and enter | sendalert victorops param.message_type=”CRITICAL” .

  6. Data API Configuration allows you to retrieve routing keys for your Splunk On-Call organization (or org-slug) via the Splunk On-Call API, which makes it easier to configure Splunk On-Call alert actions in Splunk ITSI. On this page:
    1. Enter your organization’s Splunk On-Call API ID and API Key. You can find this in Splunk On-Call’s web portal, by clicking Integrations > API. 
    2. Click the Add Key button.
    3. Click the Retrieve Routing Keys link. This retrieves the most up-to-date list of your organization’s routing keys. 

      image2.jpg

      If you have not configured your Data API keys correctly, when you configure your “Create VictorOps incident” action rule in Splunk ITSI you will get an error that reads “No Routing Keys Found”. 

  7. After your configuration is complete, you will see a check under each configuration step.

    image18.png

Next steps

You are now ready to configure action rules in the Splunk ITSI Notable Event Aggregation Policy for Splunk On-Call integration. In other words, a “Create VictorOps incident” action rule can now be used as an Alert Action in Splunk ITSI to send notifications or resolve Splunk On-Call incidents, as shown below.

Still having trouble? Splunk has many resources available to help get you back on track.

  • Splunk Answers: Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
  • Splunk Customer Support: Contact Splunk to discuss your environment and receive customer support.
  • Splunk Observability Training Courses: Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.

  • Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you require assistance.