Skip to main content
Lantern Home
 
 
Splunk Lantern

Integrating REST endpoints with On-Call

  1. Last updated
  2. Save as PDF

 

You run a small business and want to send customized alerts and incident details from your proprietary and open-source monitoring tools into the Splunk On-Call timeline.

The Splunk On-Call REST endpoint accepts alerts from any source via HTTP POST request in JSON format. Alerts are sent into the Splunk On-Call incident workflow with fields such as message_type, entity_id, or state_message. As long as you can configure the content of the request, you can trigger, acknowledge, or resolve incidents in Splunk On-Call. As these alerts come into Splunk On-Call, they are routed directly to the proper person or team with incident details that give the responder more context.

With a Splunk On-Call REST endpoint integration, you can establish incident lifecycle workflows that:

  • Send alerts straight into Splunk On-Call with detailed incident context
  • Collaborate around issues in the Splunk On-Call timeline via incident prioritization, escalation, native chat, runbook links, and more
  • Remediate incidents more quickly with automated contextual alerting, communication, and incident workflow management in one centralized location

To get started with the integration, follow these steps.

  1. In Splunk On-Call, click on Integrations > 3rd Party Integrations > REST - Generic. If the REST endpoint integration has not been enabled, click the blue Enable button to generate your endpoint destination URL.
    The REST endpoint is different from the general API and is the preferred method to create incidents.

  2. Add the routing key that will be used for this integration in the destination URL. It is essential that you replace what you see here with the actual routing key you intend to use. Everything after the final forward slash must be replaced with your key. For example, assuming a routing_key value of “database”, you would have the following: 
    ……..8177/$routing_key = => ……..8177/database

    To set up and associate routing keys in Splunk On-Call, click Settings > Routing Keys.

  3. Configure these recommended REST endpoint integration fields. Technically, only the message_type field is required. You could use {“message_type”:”critical”} as the body and generate an incident. However, in order to get the most out of the integration, we recommend including the fields outlined below. These fields help establish the workflow and lifecycle of an incident in Splunk On-Call, but keep in mind you can also include any number of additional, custom fields that you may find useful.
    Field Action

    message_type

    Behavior of the alert [String]

    Valid values:

    • CRITICAL: Triggers an incident
    • WARNING: May trigger an incident, depending on your settings
    • ACKNOWLEDGEMENT: Acknowledges an incident
    • INFO: Creates a timeline event but does not trigger an incident
    • RECOVERY: Resolves an incident

    entity_id

    ID of the Incident [String]

    This field is the identity of the incident and must remain the same throughout the life-cycle of the incident. Make sure you use the same entity_id when triggering, acknowledging, or resolving an incident. Example: disk space/db01.CSCorp.com

    entity_display_name

    Display Name in the UI and Notifications [String]

    This field allows you to give a custom, human-friendly, summary of your incidents without affecting the life-cycle workflow.

    If you are using entity_id value of dskspc/db465/<0.15volumeOnDisk/this.com, a good entity_display_name might be Disk space is below 15% on the Database Server 465. You might want to fix that.

    state_message

    Verbose message field [String]

    This field has a higher character limit and is intended for a detailed explanation of the problem. This field will be included in certain notifications (full content in email, truncated version in push/phone/SMS notifications following the entity_display_name as character limits and space allow.) Any URL links included in this field will be rendered in this field as clickable links in email notifications.

    state_start_time

    Time this issue began [Number] (Linux/Unix time)

    The time this entity entered its current state (seconds in epoch). Defaults to the time alert is received.

Expected responses

Success messages display the result and the entity_id of the incident.

 

Failure messages display the result as well as an error message explaining the reason for the failure.

Example alerts

Trigger a critical incident

This request opens a new incident.

Response:

Incident created:

Resolve an incident

This request resolves the same incident opened in the example above. Notice the entity_id is the same, allowing your system to identify which incident this request should be applied to.

Response:

Incident resolved:

Annotations

An annotation is a piece of information or link to a resource that you can add to the payload of any incident. It can be a URL, an image URL, or a note (plain text). By adding an annotation field to the alert payload, you can include annotations in an alert being sent into Splunk On-Call via the REST endpoint. INFO, WARNING, and CRITICAL alerts can be annotated by adding the annotation field to the JSON payload.

When an incident is an acknowledged state and a new alert with the same entity_id comes into the Splunk On-Call timeline, the alert is aggregated under the open incident. Any new annotations carried into the most recent alert are added to the annotation tab of the incident. The annotations are also reflected in each alert payload.

Annotation payloads

There are three types of annotations accepted by Splunk On-Call through the syntax below. You can customize the title of your annotation at the end of your annotation syntax. For example, in the below screenshot the title of the URL annotation will appear in Splunk On-Call as “Runbook”.

Example payloads

URLs

You can annotate a URL using the format vo_annotate.u.url. When annotating a URL, you must add a label, which is the text that your users will see in the payload of the incident. Clicking on the annotation will take the user directly to the URL. A common example is appending a URL to a runbook to an alert.

Image URLs

You can annotate an image URL using the format vo_annotate.i.image. Any image hosted at a URL can be rendered as an image in the payload of an incident. A common example is appending a URL link to a dashboard.

Splunk On-Call does not accept attachments and does not store images. Any images annotated to incidents must be hosted and available via URL link.

Notes

You can annotate a note using the format vo_annotate.s.note. This allows you to deliver a specific message to your users in plain text. Your annotations must fall within the 1124 character limit.

Next steps

Now that you have enabled the REST endpoint integration, you are able to send alerts from any source straight into Splunk On-Call with detailed context. Automated contextual alerting is made possible by adding in annotation field(s) with the proper syntax in the alert payload, such as runbooks, articles and dashboards to help responders triage and resolve incidents faster. Your team will be able to collaborate around issues in the Splunk On-Call timeline via incident prioritization, escalation, native chat, runbook links, and more.

If you found this article useful and want to advance your skills, Splunk Education offers a 4.5-hour, instructor-led course on Splunk On-Call Administration. The hands-on labs in the course will teach you how to:

  • Create new policies and schedules
  • Create teams and add users and managers using both the UI and API
  • Create a routing key using best practices
  • Configure Splunk On-Call integrations
  • Differentiate between the types of reports
  • Track flow of incidents after the fact using the Incident Frequency report
  • Use the Alert Rules Engine to add annotations to an incident and transform an alert
  • Create outgoing Webhooks to extend product functionality
  • Use the public API portal to find details on the public API

Click here for the course catalog where you can read the details about this and other Splunk On-Call courses, as well as register.