Maintaining adaptive thresholds

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

This task involves the maintenance of adaptive thresholds that are applied to service KPIs to ensure that they are appropriate and representative of good service function states.

This article is part of the Splunk ITSI Owner's Manual, which describes the recommended ongoing maintenance tasks that the owner of an ITSI implementation should ensure are performed to keep their implementation functional. To see more maintenance tasks, click here to see the complete manual.

Why is this important?

Adaptive thresholds are crucial in dynamically establishing baseline values for service key performance indicators (KPIs). These thresholds are essential for detecting deviations from normal behavior, indicating when an individual KPI enters a suboptimal or dysfunctional state.

The static nature of adaptive thresholds, defined for the life of the service, underscores the importance of regular tuning. Over time, the characteristics of service KPIs might evolve or drift due to changes in the IT environment, system upgrades, or shifts in user behavior. These changes can impact the baseline behavior of the monitored components, potentially leading to false positives or negatives in alerting.

Regularly revisiting adaptive thresholds allows Splunk ITSI (ITSI) users to re-train the system based on the latest insights and observations. This re-training process ensures that the adaptive thresholds remain relevant and aligned with the dynamic nature of the IT landscape. By incorporating the latest data and understanding of normal behavior, ITSI can continue to provide accurate and meaningful alerts, helping IT teams proactively address issues before they escalate.

Schedule

Every quarter

Prerequisites

Appropriate search access to the underlying Splunk environment is required to perform this activity.
This procedure requires the ITSI and Splunk Core REST APIs to be enabled and accessible from Splunk search.
This procedure requires admin access to ITSI.

Notes and warnings

When changing adaptive thresholds, it is crucial only to review services that are not currently in an error state. If a service or KPI has been outside bounds for an extended period of time, the adaptive thresholds might appear to be incorrect while being perfectly accurate.

Procedure

Step 1: Identify adaptive threshold KPIs

In the top left corner of the screen, click the Splunk logo.
On the left-hand side of the screen, select IT Service Intelligence.
In the menu at the top of the page, click Search.

Copy the following search into the search bar and press Enter to run the search:

| rest splunk_server=local report_as=text /servicesNS/nobody/SA-ITOA/itoa_interface/service fields="title,_key,kpis" 
    | spath input=value path={} output=service_object 
    | fields - value splunk_server 
    | mvexpand service_object 
    | spath input=service_object path=kpis{}.title output=KPI
    | spath input=service_object path=title output=Service 
    | spath input=service_object path=kpis{}.adaptive_thresholds_is_enabled output=adaptive_thresholding_enabled 
    | fields - service_object 
    | eval zip=mvzip(KPI, adaptive_thresholding_enabled,"&&&")
    | mvexpand zip 
    | rex field=zip "(?<KPI>.*)&&&(?<adaptive_thresholding_enabled>.*)" 
    | fields - zip
    | search adaptive_thresholding_enabled="true"
    | stats values(KPI) AS KPI BY Service

After the search has been completed, look at the results returned below the search bar. Each result represents a service KPI that uses adaptive thresholding. Remain on this page and proceed to Step 2 of the procedure.

Step 2: Review and update KPIs

Scroll to the top of the search results page and click Configuration in the menu at the top of the page.
From the subsequent dropdown menu, open Services in a new tab.
Perform the following activities for each KPI identified in the search results from Step 1, keep that tab open to refer to during this procedure.
1. Copy the Service from the search results and paste it into the Filter text box at the top of the page. Find the service in the list that appears below and click the service title.
2. In the KPIs list on the left-hand side of the page, find the KPI that corresponds with the current service from the search results and click on that KPI in the list. Then, click Thresholding in the middle of the page to see the thresholding settings for that KPI.
3. Scroll down the page to see the Preview Aggregate Thresholds visualization. This shows the fit of the current thresholding policy. Inspect how the black line interacts with the different sections of the visualization. If the black line crosses out of the green section frequently, the thresholding on this KPI may need to be re-trained. Write down the name of this Service and KPI, and then repeat the steps for the next line of the search results from Step 1.
You now have a list of KPIs that could benefit from re-training their adaptive thresholds. Instructions on how to re-apply adaptive thresholding can be found in Scenario: Apply adaptive thresholds to a KPI and detect outliers. Follow the instruction provided in this link for each KPI identified in the retraining list.

Next steps

These resources might help you understand and implement this guidance:

Splunk Docs: Scenario: Apply adaptive thresholds to a KPI and detect outliers