Moving to observability with ITSI

Glass tables

You can use the preconfigured glass tables within the Content Pack for Splunk Observability Cloud to visualize and monitor the relationships and dependencies across your IT and business services. They also allow you to create dynamic contextual views of your IT topology or business processes and monitor them in real time. You can add metrics like KPIs, ad hoc searches, and service health scores that update in real time against a background that you design, with real-time data generated by KPIs and services displayed.

Within the Content Pack for Splunk Observability Cloud there are 5 pre-configured glass tables:

1. Executive Glass Table
2. Overview Glass Table (Columns)
3. Overview Glass Table (Layered with radio gauges)
4. NOC Glass Table
5. DevOps SRE Detailed View Glass Table

Executive Glass Table

The Executive Glass Table delivers rolled-up insights across four major observability areas: Synthetic Monitoring, Infrastructure Monitoring, Splunk Real User Monitoring, and Application Performance Monitoring. In a single view, you get a quick and real-time update as to what is going on across all of your landscape and single-click access to additional details. This answers the question of what is going on and why it’s happening, and provides a contextual deep-link to the source system in Splunk Observability Cloud. This dramatically improves MTTI, MTTD, and MTTR, by simplifying the process of finding and resolving the root cause issue quickly.

In the bottom left corner, you will also find key trends: Total, Splunk Synthetic Monitoring, Splunk Application Performance Monitoring, Splunk Infrastructure Monitoring and Splunk Real User Monitoring. This makes it easy to understand the history across your applications, infrastructure and customer experiences with a simple glance.

Overview Glass Table (Columns)

This glass table provides a quick overview of the Splunk Synthetic Monitoring, Splunk Infrastructure Monitoring, and Splunk Application Performance Monitoring .

After you click into the details, with another click you move into the Splunk Observability Cloud with an embedded deep link, providing you with the native results fully in context.

Overview Glass Table (Layered with radio gauges)

This glass table provides immediate insight and results, with the radio gauges on the left side making it simple to know where Splunk Synthetic Monitoring, Splunk Infrastructure Monitoring, and Splunk Application Performance Monitoring results are for your entire environment.

These types of insights deliver immediate value to a variety of teams and stakeholders. You can click on any of these KPIs, and get to the details, and click again to get to the Splunk Observability Cloud native results fully in context.

In the example glass table shown below, you might first click into Splunk Synthetic Monitoring to see what is going on, but you can also see there is a problem with AWS in the Infrastructure Monitoring Health Summary. That could be a root cause of the Splunk Synthetic Monitoring problem and perhaps a first place to investigate.

NOC Glass Table

The modern NOC (Network Operations Center) is evolving to more observability and increased awareness of what is going on across all stacks within their users, infrastructure, critical business flows, and applications.

An interactive glass table gives results in real-time, helping the NOC team to proactively manage situations or episodes as they occur, and quickly find and resolve issues.

This simplified view is designed for a wall or hallway monitor. It shows the trend and history along with current state for all your critical KPIs, rolled up for visibility and awareness.

DevOps SRE Detailed View Glass Table

Here you gain insights across Splunk Synthetic Monitoring, Splunk Infrastructure Monitoring, Splunk Application Performance Monitoring, and Splunk Real User Monitoring. Insights are available not only at a summary level, but also two levels down, with single-click access to additional details for each. 

Service Analyzer

The Service Analyzer is your starting point for monitoring your IT operations, enabling you to see the live health of your IT environment at a glance.

The Service Analyzer provides an overview of ITSI service health scores and KPI search results that are currently trending at the highest severity levels. You can use the Service Analyzer to quickly view the status of IT operations and to identify services and KPIs running outside expected norms. Click any tile in the Service Analyzer to drill down for further analysis and comparison of search results over time.

There are two Service Analyzer views: the tile view and the tree view. You can drill down to more detailed information from each view to investigate services with poor health scores.

Within this Service Analyzer you are viewing in tree view, and you can clearly see each of the four services and the status of how the underlying KPIs are reporting.

Services

A service is a logical mapping of IT objects that applies to your business goals. The definition of a service is fairly broad. You can create business and technical services that model those within your environment, with some services that might have dependencies on other services. Services contain KPIs which make it possible to monitor service health, perform root cause analysis, receive alerts, and ensure that your IT operations are in compliance with business service-level agreements (SLAs).

In the example below, you can see the tile view of the Service Analyzer. This provides you with insights across all 25+ services and their status for the time range selected, along with the ability to click into more results for any of these services to see the KPIs, entities, and more.

KPIs

A key performance indicator (KPI) is a recurring saved search that returns the value of an IT performance metric, for example CPU load percentage, memory used percentage, or response time. A KPI is used to monitor the health of a service.

You create KPIs within a specific service. The KPI allows you to generate searches to help you understand the underlying data, including how to access, aggregate, and qualify with thresholds. You can use the search results to monitor service health, check the status of IT components, and troubleshoot trends that might indicate an issue with your IT systems.

The Content Pack for Splunk Observability Cloud provides over 80+ KPIs, giving you deep insights across your observability results. The example below shows how access to the results are a single click away. You can also quickly see the underlying entities and how each is reporting.

Entity types

The Content Pack for Splunk Observability Cloud includes custom entity types. You can use associations to visualize and troubleshoot various entities. For example, the Content Pack comes with an entity type of “AWS EC2” to import your AWS EC2 services as entities. You can group entities by entity type in the Infrastructure Overview, enabling visualization of key metrics relating to the health of AWS EC2 entities (or services).

The content pack includes 15 custom entity types, one for each of the metrics from the Splunk Infrastructure Monitoring Add-on, one for Splunk APM, and one for each of the metrics from the Splunk Synthetic Monitoring Add-on.

1. AWS EC2
2. AWS Lambda
3. Azure Functions
4. Azure VM
5. GCP Cloud Functions
6. GCP Compute Engine
7. Kubernetes Pods
8. OS Hosts
9. Splunk Infrastructure Monitoring
10. Splunk APM
11. Synthetic API
12. Synthetic Benchmark
13. Synthetic Content
14. Synthetic HTTP
15. Synthetic Real Browser

Vital metrics

Within the Content Pack for Splunk Observability Cloud, you will receive 34 vital metrics out of the box. These show a critical summary within the entity type via a set of vital metrics which describe the overall health of entities of that type, including things like: CPU Utilization, Network In, Network Out, Disk Read Ops, Disk Write Ops, and more. You can view these metrics on the Entity Health page and drill down further into individual exchange entities. 

You can optionally add, modify, or delete the preconfigured entity types. For instructions to create and edit entity types, see Create custom entity types in ITSI. 

Dashboards

Dashboards are used to display tables or charts in panels which hold the summarized data in a visually appealing manner. You can add multiple panels and multiple reports and charts to the same dashboard, providing easy access to valuable information in a quick and easy manner.

Within the Content Pack there are 13 different dashboards:

1. SIM - Infrastructure Metrics and Logging
2. Splunk APM Overview
3. Splunk Infrastructure Monitoring Command Health Check
4. Splunk Infrastructure Monitoring Modular Input Health Check
5. Synthetic API Check Detail
6. Synthetic Benchmark Check Detail
7. Synthetic Content Check Detail
8. Synthetic HTTP Check Detail
9. Synthetic Monitoring KPI Browser
10. Synthetic Monitoring KPI Comparison
11. Synthetic Real Browser Check Detail
12. Welcome to Splunk Synthetic Monitoring Add-on
13. SIM Navigation

The example below shows the Splunk APM Overview dashboard, providing a RED Metrics (Rate, Error, Duration) summary. In this example, you are looking at the “checkout” service.