Configuring notable event timestamps to match raw data
When reviewing notable events in the Event Timeline of the episode view, you might notice some irregularities such as the timestamp of the notable event not matching what you see in the upstream monitoring tool, or two notable events sharing the same time, as seen in the screenshot below.
Further, when you look at the raw data, you see that the events happened at different times, 10:46 and 10:48 in the following screenshot. However, as notable events, the time for both shows as 10:49.
You need to fix both of these issues so that you can be properly notified of notable events and have accurate information about them.
This article is part of the Definitive Guide to Best Practices for IT Service Intelligence. ITSI end users will benefit from adopting this practice as they work on Event Analytics.
Solution
The situation above is caused by the notable events inheriting the timestamp of the correlation search that created them rather than the timestamp of the underlying raw data. To fix this, you need to configure the is_use_event_time parameter in SA-ITOA/local/alert_actions.conf.
If you are a Splunk Cloud Platform customer, you need to file a support ticket to set this parameter.
> $SPLUNK_HOME/opt/splunk/etc/apps/SA-ITOA/local/alert_actions.conf
[itsi_event_generator] param.is_use_event_time = 1
Now notable events use the timestamp of raw alert, and your UI more effectively shows what is happening in your environment, as seen in the following screenshots.
Next steps
You might also be interested in the following Splunk resources:
- Splunk Docs: Event analytics manual
- Splunk Docs: ITSI administrators manual: alert_actions.conf