When reviewing notable events in the Event Timeline of the episode view, you might notice some irregularities such as the timestamp of the notable event not matching what you see in the upstream monitoring tool, or two notable events sharing the same time, as seen in the screenshot below.
Further, when you look at the raw data, you see that the events happened at different times, 10:46 and 10:48 in the following screenshot. However, as notable events, the time for both shows as 10:49.
You need to fix both of these issues so that you can be properly notified of notable events and have accurate information about them.
This article is part of the Definitive Guide to Best Practices for IT Service Intelligence. ITSI end users will benefit from adopting this practice as they work on Event Analytics.
The situation above is caused by the notable events inheriting the timestamp of the correlation search that created them rather than the timestamp of the underlying raw data. To fix this, you need to configure the is_use_event_time parameter in SA-ITOA/local/alert_actions.conf.
If you are a Splunk Cloud Platform customer, you need to file a support ticket to set this parameter.
[itsi_event_generator] param.is_use_event_time = 1
Now notable events use the timestamp of raw alert, and your UI more effectively shows what is happening in your environment, as seen in the following screenshots.
This content comes from the .Conf23 session, The Definitive List of Best Practices for Splunk® IT Service Intelligence: How to Configure, Administer, and Use ITSI for Optimal Results. In the session replay, you can watch Jason Riley and Jeff Wiedemann share the many awesome best practices they've amassed for designing key performance indicators (KPIs), services, episodes, and machine learning to maximize end-user experience and insights. Whether you're new or experienced, you'll come away with tactical guidance you can use right away.
You might also be interested in the following Splunk resources: