Pushing alerts to the Splunk platform and ITSI
Splunk ITSI (ITSI) has a robust event analytics engine that can perform a variety of functions. To remediate events quickly, you should push alerts to the Splunk platform, rather than relying on a scheduled input to pull events.
This article is part of the Definitive Guide to Best Practices for IT Service Intelligence. ITSI administrators will benefit from adopting this practice as they work on Event Analytics.
Solution
All episodes follow a basic pattern:
- Something being monitored fails.
- The monitoring system detects the failure.
- An alert is created.
- The situation is resolved in some manner.
- The monitoring system detects that the condition has cleared.
- A clearing alert is created.
So what is the difference in this pattern between pushing these events to the Splunk platform versus waiting for the Splunk platform to pull them? The following two timelines show a comparison.
The Splunk platform waits for alerts to be pulled
This scenario introduces lag in several stages. There can be a significant amount of time between when the alert is created in ITSI and when the scheduled input runs. The same lag occurs when the situation is resolved. So while a service might have only been down for a minute, ITSI might think it was down for a half hour. During that lag, you might be paging people to work on a situation that has already been resolved.
An alert is pushed the Splunk platform
In this scenario, events are pushed to the Splunk platform. Doing this creates and clears the episode almost immediately. There are fewer steps and they are completed in less time.
Next steps
You might also be interested in the following Splunk resources:
- Splunk Docs: Event analytics manual
- Splunk Docs: Send data to Splunk Cloud Platform with ITSI data collection agents
- Splunk Docs: Best practices for implementing Event Analytics in ITSI