Skip to main content
 
 
 
Splunk Lantern

Configuring the ITSI Notable Event Aggregation Policy

 

​You want to configure and enable the Notable Event Aggregation Policy (NEAP) to process notable events so they can be grouped into meaningful Splunk ITSI episodes.

Before following these steps, make sure you have done the following:

Solution

The diagram below shows the overarching architecture for the integration that's described in Managing the lifecycle of an alert: from detection to remediation. The scope for this article is indicated by the pink box in the diagram.

process to create NEAP policies in ITSI for Splunk On-Call during the lifecycle of an observability alert

In this article, you'll learn how to configure the Notable Event Aggregation Policy (NEAP) within the Content Pack for ITSI Monitoring and Alerting. The NEAP does two things:

The resulting episodes are stored in a Splunk index called ITSI_Grouped_Alerts. The episodes can reduce alert noise significantly since although the alerts might be initiated from Splunk Real User Monitoring, Splunk Synthetic Monitoring, or Splunk Application Performance Monitoring, you end up with one actionable episode rather than many individual events.

Configuration

  1. If you haven't already, download this ITSI Backup file and use the ITSI Backup/Restore utility to restore the artifacts into your instance of Splunk ITSI.
  2. On the Notable Event Aggregation Policies page, find "Episodes by Application/SRC o11y" and click it.Notable event aggregation policies
  3. The first tab, as seen below, is the "Filtering Criteria and Instructions". This defines what notable events are being evaluated. Note the first grouping below where certain notable events are excluded. This is specific to this solution since you don’t want to evaluate Splunk ITSI service health or KPI-generated notable events. You only want to evaluate events coming in from your Splunk Observability Cloud alert detectors.

    Filtering Criteria and Instructions notable events are being evaluated. The first being excluded.

  4. Scroll down and review other policy settings. Specifically note the following two:
    1. Split events by field. This is set to app_name. This represents metadata from Splunk Observability Cloud that identifies the application.
    2. Episode information > Episode Title. The %% fields will be resolved as variables or metadata coming over from a Splunk Observability Cloud alert (detector).

      Split events by field, break episode and episode information. Configuring episode dashboard by episode title, description, severity, assignee, status and instructions.

  5. Click the Action Rules tab to review the actions configured to create the Splunk On-Call (VictorOps) incident, as well as the closing of the Splunk On-Call incident when the Splunk Observability Cloud alert is cleared.
    1. The first rule needs a source name of "Episode Monitoring - Trigger OnCall Incident" within the notable alerts (itsi_tracked_alerts index). The source correlates to the originating correlation search, which indicates it is time to create a new Splunk On-Call incident. When found, a comment is added and a call to the "Create VictorOps Incident" integration is performed.

      Action rules are defined as rules to take automated actions on an episode when specific activation criteria are met

    2. The second rule needs a source name of "Episode Monitoring - Set Episode to Highest Alarm Severity o11y" within the notable alerts (itsi_tracked_alerts index). The source correlates to the originating correlation search, which indicates it is time to close the Splunk On-Call incident. The rule also needs the "set_episode_status" to equal 5. When both equate to true, then the episode status is changed to "Closed", a comment is added to indicate closure, and the "Create VictorOps" integration is called with a value of "RECOVERY" to indicate the incident should be closed. This allows for synchronization between the Splunk ITSI episode and the Splunk On-Call incident.

      If a specific even occurs, then change status to Closed for the episode, and add a comment for the episode and perform the selected action for the episode

Next steps

Now that you’ve successfully reviewed the configuration of the Splunk ITSI Notable Event Aggregation Policy (NEAP), continue to the next article to configure ITSI correlation searches for monitoring episodes.

Still having trouble? Splunk has many resources available to help get you back on track.

  • Splunk Answers: Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
  • Splunk Customer Support: Contact Splunk to discuss your environment and receive customer support.
  • Splunk Observability Training Courses: Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.
  • Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you would like assistance.