Skip to main content
Lantern Home
 
 
Splunk Lantern

Normalizing Observability Cloud alerts into the ITSI Universal Alerting schema

  1. Last updated
  2. Save as PDF

 

You have integrated your Splunk Observability Cloud alerts with Splunk Cloud Platform or Splunk Enterprise and now need to normalize the data to ensure it adheres to the ITSI Universal Alerting schema. This ensures that the Universal Correlation Search you'll create later can create notable events in Splunk ITSI from your Splunk Observability Cloud events.

Solution

The diagram below shows the overarching architecture for the integration that's described in Managing the lifecycle of an alert: from detection to remediation. The scope for this article is surrounded by the pink box in the diagram.

Procedure

  1. Install the Content Pack for ITSI Monitoring and Alerting.
  2. Review the Universal Alerting section of the Content Pack Docs.
  3. Go to the Search and Reporting app in Splunk Cloud Platform or Splunk Enterprise and open a search window.
  4. Run the following search: index="alerts" source=o11y_alerts_webhook . Your search results should look like this:Screenshot 2023-05-15 at 3.29.04 PM.png
  5. Access the Field Knowledge Object window in the Splunk platform user interface by clicking Settings > Fields > Calculated Fields.
  6. To add each eval entry, click the New Calculated Field button. The following form opens:image4.png
    1. Enter itsi in Destination app.
    2. Enter source in Apply to.
    3. Enter o11y_alerts_webhook in named.
    4. Enter the first calculated field shown in the table below in the Name and Eval expression fields. Then, save the form, open a new one, and repeat for every remaining row in the table, using the same first three values in the Destination app, Apply to, and named fields.
      Name Eval expression
      app O11yWebhook
      itsiDrilldownURI detectorUrl
      itsiInclude

      “false"

      You will change this field to true at the end of the process, after all the intended fields have been added and tested.
      severity_id case(status="ok",2, severity="Critical", 6, severity="Major", 5, severity="Minor", 4, severity="Warning", 3, severity="Info", 1, 1=1, 1)
      signature spath(_raw, "eventType")
      src case(isnotnull('inputs.attempt_condition.key.sf_service'),'inputs.attempt_condition.key.sf_service',isnotnull('inputs.A.key.k8s.cluster.name'),"K8S Cluster: " . 'inputs.A.key.k8s.cluster.name',1=1, src)
      status_o11_alert status
      subcomponent case(isnotnull('inputs.attempt_condition.key.sf_operation'), 'inputs.attempt_condition.key.sf_httpMethod' . 'inputs.attempt_condition.key.sf_operation',isnotnull('inputs.A.key.k8s.cluster.name'), "K8S Pod: " . 'inputs.A.key.k8s.pod.name', 1=1, subcomponent)
      vendor_severity severity
  7. In Calculated Fields, click All under App and enter o11y_alerts_webhook in the search field.
  8. Review the calculated fields created from the previous step:image1.png
  9. Go to the Search and Reporting App search window and run the following search: index="alerts" source=o11y_alerts_webhook
  10. Verify the extractions were created and the values look correct.image6.png
  11. Finally, you need to change the value of the new field itsiInclude from false to true. Go back to Calculated Fields to do this, search "itsiInclude", click the name of the field to edit it and change the value, then click Save. If the Universal Correlation Search in Splunk ITSI is enabled, alerts from this new source will now be ingested as notable events. 

Next steps

Now that you’ve successfully normalized the Splunk Observability Cloud alerts into the Splunk ITSI universal alert schema, continue to the next article to Configure ITSI correlation search to create notable events.

Still having trouble? Splunk has many resources available to help get you back on track.

  • Splunk Answers. Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
  • Splunk Customer Support. Contact Splunk to discuss your environment and receive customer support.
  • Splunk Observability Training Courses. Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.

  • Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you require assistance.