Correlating log data to metric charts in Observability Cloud dashboards

Use different types of log charts in Splunk Observability Cloud

There are two types of log charts supported in the Splunk Observability Cloud dashboards:

• Log Views. This chart shows a readable view of logs and their metrics.
  
• Log Timeline. This chart shows the rate of logged events over time, grouped by the fields and values of your choice.
  

Use log charts to find the root cause of issues

You can drill into specific logs from the log charts to determine root cause quickly, for example, where an SRE is seeing a spike in error rate for a payment service.

1. Open the service dashboard and look at the payment service. In the example below, the LogErrorsAggregate chart on the right shows the number of error log events, and the detailed logs are visible underneath this.

   

2. Click one of the detailed logs to examine it. The screen displays all of the details of the individual event and allows you to navigate to the error detail in the relevant time period. In the screenshot below, the arrow points to the message where an invalid token was used, showing the root cause.

   

Use charts in an Splunk Observability Cloud dashboard using the Splunk Log Observer UI

Both types of charts can be easily configured and saved as charts in an Splunk Observability Cloud dashboard using Splunk Log Observer Connect.

1. Navigate to the Splunk Log Observer icon on the left UI navigation panel.

   

2. Configure the chart views you need. In this example you'll configure the Log Timeline chart. Configure the appropriate source of log events by selecting the o11y-1 Log Observer Connect integration and the otel_events index. These are logs coming from the Splunk Observability Cloud Kubernetes Open Telemetry collector that sends logs to the Splunk platform instance.

   

3. Enter severity, service.name in the Group by field. This provides the count by severity, with service.name as the aggregation. You'll need to include service.name grouping so the filtering will occur in the new dashboard chart you create.

   

4. Add a filter by clicking Add Filter at the top of the UI. Click α Fields, and then click error. This changes the count aggregation to only include logs with a severity that equals error.

   

   The results look like this:

   

5. In the Save drop-down menu in the top right of the UI, click Save to Dashboard.

   

6. Enter the following details:
   1. Enter the name of the new chart.
   2. Enter an optional description.
   3. Select the dashboard where you want to create the new timeline log chart.
   4. Select Log timeline as the log chart type.
   5. Click >Save and go to dashboard.

      

The chart is created in the dashboard like this:

To view or modify the chart, click ... in the chart and select View in Log Observer.