Checking for KPI search success

Step 1: Identify the percentage of KPI searches being skipped

1. In the top left corner of the screen, click the Splunk logo.
2. On the left-hand side of the screen, select IT Service Intelligence.
   
3. In the menu at the top of the page, click Search.
   
4. Copy the following search into the search bar and press Enter to run the search:

   index=_internal sourcetype=scheduler savedsearch_name="Indicator*" earliest=-7d@d latest=now()
   | stats dc(sid) AS run_count, count(eval(status="delegated_remote_error" OR status="skipped")) AS failed_count
   | eval skip_ratio=failed_count/run_count*100
   | rename run_count AS "KPI Searches Run" failed_count AS "KPI Searches Failed" skip_ratio AS "KPI Search Failure Percentage"

5. Review the search results in the panel below the search bar. Look at the value in the KPI Search Failure Percentage column.
   • If this value is greater than 0, there is a problem with searches running in ITSI that needs to be investigated further. Move on to Step 2 below.
   • If that value is 0, this task is complete.
   

Step 2: Investigate KPI search skip rate further

Further investigation is required if the failure percentage identified in the previous step is greater than 0.

1. Copy the following search into the search bar and press the enter key on your keyboard to run the search:

   index=_internal sourcetype=scheduler savedsearch_name="Indicator*"  earliest=-7d@d latest=now()
               | stats dc(sid) AS run_count, count(eval(status="delegated_remote_error" OR status="skipped")) AS failed_count, max(run_time) AS max_runtime, earliest(_time) AS first, latest(_time) AS last BY savedsearch_name
               | eval KPI_search_type=if(savedsearch_name like "%Shared%", "base", "ad hoc")
               | eval runtime_headroom_pct=round((100-(max_runtime/((last-first)/(run_count-1))*100)),1)
               | join type=outer savedsearch_name [search index=_internal sourcetype=scheduler savedsearch_name="Indicator*"
               | join sid
               [ search index=_internal sourcetype=splunk_search_messages app="itsi" log_level=ERROR]
               | stats count(savedsearch_name) AS "error_count" by savedsearch_name]
               | rex field=savedsearch_name ".* (?<base_id>.*?) - ITSI Search$"
               | lookup itsi_kpi_attributes kpiid AS base_id OUTPUTNEW kpi_name service_name
               | lookup kpi_base_search_title_lookup _key AS base_id OUTPUTNEW title AS kpi_name
               | eval ri=if(runtime_headroom_pct<25,"Yes","No")
               | eval sf=if(failed_count>0,"Yes","No")
               | eval se=if(error_count>0,"Yes","No")
               | search ri="Yes" OR sf="Yes" OR se="Yes"
               | fields KPI_search_type service_name kpi_name ri sf se
               | rename sf AS "Search Failures (More than 0 failures)", ri AS "Runtime Issues (Less than 25% run time headroom)", se AS "Search Errors (More than 0 Errors)"

2. Review the results from the search. Each row identifies a KPI search that has a problem. The search shows the type of KPI (base search or ad hoc search), the associated service for ad hoc searches and the KPI name to allow for easy identification of the KPI in the ITSI interface. The last three columns in the search results indicate the detected problem for each KPI search. The three problems detected by this search are:
   • KPI searches with failures in execution (Search Failures column)
   • KPI Searches that are taking too long to run based on their schedule (Runtime Issues column)
   • KPI Searches that have search errors flagged by the Splunk platform (Search Errors column)
   
3. Investigate every KPI search identified in these search results in depth to identify the root cause of the issue and be remediated. If you need assistance with this task, please contact your Splunk account team to engage professional services support.