Skip to main content
Lantern Home
 
 
Splunk Lantern

Choosing KPI base searches over ad hoc searches

  1. Last updated
  2. Save as PDF

 

Some of your KPIs display N/A instead of a value on your dashboards in Splunk ITSI. You need to resolve this so you can get KPI and Health Score values. 

This article is part of the Definitive Guide to Best Practices for IT Service Intelligence. ITSI administrators will benefit from adopting this practice as they work on Service Insights. 

Solution 

Multiple KPIs can be powered by the same dataset to dramatically reduce search load and reduce unknown KPI values. Using base searches results in one search for multiple KPIs, which means you run fewer searches and use less time, so there is more capacity for additional searches. If you don't use base searches, each KPI runs on its own schedule and uses up more capacity. 

For example, you might be able to create a base search that powers four KPIs and runs 129 times per day. If you had an ad hoc search for each of those KPIs, that would mean 516 searches running each day. Search capacity is limited by the number of search slots that are available. If a KPI search cannot execute due to search capacity constraints, it displays N/A for an unknown value. This unknown value interferes with service Health Score calculation, which then leads to potentially unreliable actions. The base search reduces searches being run by 75 percent.

clipboard_e300a1d8ee7ca66d67f66f7fddb1da6c0.png

In addition, base searches help KPIs stay in sync. When KPIs run at different times, you get out-of-sync values, which can also lead to unreliable health scores. In this example, ad hoc searches run at different times give an incomplete picture. The health score of the service is 100, but we only see a total value of 93 when looking at the KPIs.

clipboard_ee4699b84a044724b3ed9181c4e7a9b09.png

In this second example, when base searches have been applied, all 100 stations are accounted for.

clipboard_ee18383d204bc1b94438068403657b91f.png

Next steps

This content comes from the .Conf23 session, The Definitive List of Best Practices for Splunk® IT Service Intelligence: How to Configure, Administer, and Use ITSI for Optimal Results. In the session replay, you can watch Jason Riley and Jeff Wiedemann share the many awesome best practices they've amassed for designing key performance indicators (KPIs), services, episodes, and machine learning to maximize end-user experience and insights. Whether you're new or experienced, you'll come away with tactical guidance you can use right away.

You might also be interested in the following Splunk resources: