Skip to main content
Lantern Home
 
 
Splunk Lantern

Reviewing your ITSI environment

  1. Last updated
  2. Save as PDF

 

Because of the complexities in changing IT environments, over time you might find that your Splunk ITSI environment is not performing as expected. You want to perform periodic checks to make sure that it is healthy and to find ways to optimize it.

This article is part of the Definitive Guide to Best Practices for IT Service Intelligence. ITSI end users will benefit from adopting this practice as they work on Service Insights. 

Solution 

First, outside of Splunk ITSI, review your services to make sure the architecture hasn't changed, or to make sure any changes are reflected in your environment. After you have done that, use the ITSI Health Check Dashboard, the Event Analytics Monitoring Dashboard, and the Cloud Monitoring Console to verify that Splunk ITSI is healthy.

ITSI Health Check Dashboard

  1. Check your ratio of base searches to ad hoc searches.

    clipboard_e84277ba43160c0010e4cc1f7c7b02bef.png

  2. Look at your KV store. Is it growing too large and unmanageable? This is especially a concern in a cluster environment where you can start to run into replication issues.

    clipboard_e77c60dc8464e5ce8f1055e2a9c7111e8.png

  3. In the KPI Performance panel, look at the runtime headroom for your searches. If your KPI is set to refresh every 5 minutes, but your search is so dense that it takes 10 minutes to execute, that's not going to produce meaningful value. Headroom lets you see the searches that aren't completing in the right amount of time.

    clipboard_ede2459de712cc437ad6df29dafbb3a92.png

  4. Look at the Concurrent Searches panel to see if searches are being skipped.

    clipboard_e266b94d6e4c1c86feb5eebf2add7c970.png

Event Analytics Monitoring Dashboard

Select the Skipped Events panel to see whether skipped events to make sure they are being aggregated correctly. The Event Time Processing Panel will tell you how long it takes them to run and help you determine whether you need to make adjustments.

clipboard_e4394d24cf97486f773b1062fc3c50378.png

Cloud Monitoring Console - Scheduler Activity Dashboard

If you are a Splunk admin (not an ITSI admin), you can use the Cloud Monitoring Console to look at the skip ratio for searches. This can help you determine whether any KPIs need to be refactored.

clipboard_e1a1583f98664cf26557a70602e79b6b5.png

You can also dig into detail about skipped searches.

clipboard_e42d437285f401df8e5b8ea3ef9a0b057.png

Next steps

This content comes from the .Conf23 session, The Definitive List of Best Practices for Splunk® IT Service Intelligence: How to Configure, Administer, and Use ITSI for Optimal Results. In the session replay, you can watch Jason Riley and Jeff Wiedemann share the many awesome best practices they've amassed for designing key performance indicators (KPIs), services, episodes, and machine learning to maximize end-user experience and insights. Whether you're new or experienced, you'll come away with tactical guidance you can use right away.

​You might also be interested in the following Splunk resources: