Managing the lifecycle of an alert: from detection to remediation
Many organizations experience significant challenges with the alerting process, ranging from overwhelming volumes of alerts leading to fatigue and missed critical incidents, to a lack of contextual information that prevents effective prioritization and response. Inadequate workflows and communication channels can also prevent efficient alert triage and resolution.
To overcome these hurdles, it's important to adopt a comprehensive approach to managing the lifecycle of an alert, encompassing detection, triage, investigation, and remediation. An effective workflow can help you to drive significant improvements in mean time to detect (MTTD) and mean time to respond (MTTR) to incidents that can cause you significant operational issues.
This use case spans a number of different Splunk products that work together to produce a complete alert management workflow. The workflow uses events generated in Splunk Observability Cloud, making them available for use in Splunk ITSI. These alerts are normalized, sorted, and grouped into episodes that can be seen through Episode Review in Splunk ITSI. Splunk On-Call is also integrated so that when incidents occur, the right teams can be quickly notified. This workflow is demonstrated in the following image.
For a comprehensive overview of the event analytics workflow as it relates to Splunk ITSI, read the Splunk Docs Overview of Event Analytics.
Data required
Prerequisites
- Download and install the Content Pack for ITSI Monitoring and Alerting. The preconfigured correlation searches and notable event aggregation policies in the content pack help you to produce meaningful and actionable alerts.
- Download the ITSI Backup file and use the ITSI Backup/Restore utility to restore this into your version of Splunk ITSI. The backup file contains a number of modified correlation searches and notable event aggregation policies that are used in the workflow laid out below.
How to use Splunk software for this use case
The articles in this use case are intended to flow and build upon each other, although they can stand on their own for a specific capability you might be interested in.
Next steps
This event analytics and incident management configuration and design framework helps drive operational excellence and value through significant improvements in mean time to detect (MTTD) and mean time to respond (MTTR) to the incidents that can cause you significant operational issues.
Still having trouble? Splunk has many resources available to help get you back on track.
- Splunk Answers: Ask your question to the Splunk Community, which has provided over 50,000 user solutions to date.
- Splunk Customer Support: Contact Splunk to discuss your environment and receive customer support.
- Splunk Observability Training Courses: Comprehensive Splunk training to fully unlock the power of Splunk Observability Cloud.