Prescriptive Adoption Motion - Infrastructure Monitoring

Other use cases to consider

Use cases are specific to each organization, so also consider these thought starters as a way to help with your ideation of new use cases. Be sure to monitor availability, performance, capacity, and error conditions for any new components.   

• Hosted services monitoring
• Database monitoring 
• Network monitoring
• Serverless function monitoring
• Middleware monitoring
• Virtualization platform monitoring
• Storage monitoring
• Identity and access management (IAM) monitoring

1. Prerequisites

To get data into Splunk Observability Cloud for infrastructure monitoring:

• You must be an administrator in Splunk Observability Cloud.
• You must have an access token for the Splunk Observability Cloud organization you want to get data into.
• If you are connecting to Amazon Web Services, you must have an access token for the Splunk Observability Cloud organization you want to get data into.

2. Recommended Training

• Introduction to Splunk Observability  - free 
• Introduction to Splunk Infrastructure Monitoring  - free
• Visualizing and alerting in Splunk Infrastructure Monitoring - 4.5 hour virtual or onsite
• Automation using the REST and SignalFlow API - 2 day virtual or onsite
• Using the Splunk IM Terraform provider - 2 day virtual or onsite
• Ingesting application metrics in Splunk Observability Cloud - 4.5 hour virtual 
• Kubernetes monitoring with Splunk Observability Cloud - 4.5 hour virtual 
• Network performance monitoring with Splunk Network Explorer - free

3. Resources

• Professional Services - Observability
• On-Demand Services - Infrastructure Monitoring
• Assigned Expert (AE)

4. Considerations

Observability is a broad concept, and organizations can have difficulty determining what to monitor. But many organizations also find that monitoring four key capabilities help to monitor the aspects of your systems in a way that meets the needs of your organization: availability monitoring, error conditions monitoring, performance monitoring, and capacity monitoring. When a full 360-degree view of an end-to-end production system is needed, observability should be instrumented for all four capabilities. Failing to monitor all four elements of a live production system increases the risk of business/operation disruption and significantly reduces time to restore service should a disruption occur.

Whether your objective is to horizontally monitor all of the same type of a set of common server types (for example, all Linux servers or Kubernetes pods) or to monitor only the infrastructure that supports a critical business application (for example, the Linux or Windows databases), Splunk recommends that you employ the capabilities in the following sequence to achieve that objective.

1. Availability monitoring. Monitoring availability is the foundational observability element. You must monitor for the very existence of the object being observed because if something doesn’t exist, nothing else matters. You must know if an application is running and if the intended users can use it. In the physical world, your favorite store must be open so that you can walk in and buy something. If it is closed (that is, unavailable), nothing else about its operations matters.
2. Error conditions monitoring. Error conditions are meant to be a catch-all concept. There are infinite situations that can determine if a system has an issue that is currently disrupting the business or is imminently going to cause disruption if there is no intervention. It is generally intended to look at logs for messages that indicate a problem has occurred or is about to occur. The error conditions element is where ‘state monitoring’ fits. State monitoring is when something goes from one state to another such as from on to off, or from up to down.
3. Performance monitoring. After you have a good implementation of monitoring availability, it is important to understand how well it is performing. Think again about your favorite physical store. When the store is available and shoppers are shopping, you might want to understand the duration it takes a shopper to proceed through the checkout line. This is the same for systems; you want to know the time it takes for a user to receive a checkout complete message from the time they clicked on the pay now button. This is just one of many different types of performance measures, but they are all about cycle time, or how long it takes to complete a task.
4. Capacity monitoring. Capacity is the resources that are consumed for a particular function. Capacity can come in many different measurements such as, bandwidth, storage space, memory utilization, number of purchases executed simultaneously, and more. If a physical store only has one cash register and a marketing promotion drives a large number of people to shop there at the same time, the duration to proceed through the checkout line could increase dramatically. The same is true in systems. Lots of people using the system at the same time can cause really slow response times or possibly crash a system.

You might choose to fully implement one capability before moving to the next. Some organizations have employed an iterative strategy in which they do partial instrumentation for each and then cycle back through two or three more times, adding additional metrics, logs, and traces each time. Your approach should be taken based on the particular needs of your organization.