Review alerts received when a pending state occurs
You work for a large organization that uses the Kubernetes container management platform. When a pending state is detected and an alert is sent, you need your team to be able to easily review the alert details and get involved immediately so that they can take action straight away.
How to use Splunk software for this use case
Alert messages can be delivered to various sources including email, teams, and webhooks. Slack is used as the alert message channel of choice in this example.
- Review the alerts received from Splunk Observability Cloud and Splunk Incident Management. In this example, a Slack message has been received from Splunk Incident Management through the
#sre_notificationschannel indicating an incident has occurred that needs to be acknowledged, and a corresponding Splunk Observability Cloud alert has also been generated. From the alert you can:- Acknowledge, Resolve, Reroute, or Snooze the alert.
- View the alert without taking any action by clicking the alert title link in the Slack message, which takes you to the alert in Splunk Observability Cloud.
If you receive the alert notification through a different channel, for example through email, you can click a link in the alert message to view the alert details in Splunk Observability Cloud.
-
In Splunk Observability Cloud, review the alert details to determine which pod caused the alert. You can also use the Explore Further section links to open Splunk Application Performance Monitoring, Splunk Synthetic Monitoring, view the data links, or manage the alert.
- As part of your investigation, you determine that a new payment service deployment caused the issue and needs to be rolled back. To verify it has been rolled back and is no longer an issue, click Infrastructure in the left navigation pane and click K8s workloads.
- Search for the payment service workload and verify the pod is no longer in a pending state.
- You can now set the alert as resolved by clicking Resolved.
Next steps
These additional Splunk resources might help you understand and implement these recommendations:
- Splunk Training: Visualizing and alerting in Splunk IM
- Splunk Training: Kubernetes monitoring with Splunk Observability Cloud