Implementing features and use cases in Splunk APM

Indexing span tags

Drill down into service performance with span tags. Span tags provide additional context about operations that spans represent. Default span tags include things like the endpoint, operation, and HTTP method associated with a span. Using these tags, you can analyze requests, errors, and latency for spans that contain specific span tags. This context lets you understand service performance at a glance and helps you discover the root cause of issues faster.

Index span tags to analyze services in the following ways:

• Break down service performance by indexed tags in the Troubleshooting Service Map.
• View charts of service performance metrics by indexed span tags in Tag Spotlight.
• Track multiple traces for a specific activity with Business Workflows.

► Which span tags to index, and how to index span tags

Index only span tags you want to drill down into to gain insights about the performance of your infrastructure, or to address a specific incident. Some span tags provide a level of cardinality that just isn’t useful. For example, indexing query_id would generate MetricSets for every unique query, and in most cases there’s no reason for this level of cardinality. Also, avoid indexing span tags that represent ephemeral resources, such as container_id.

Here are the span tags that APM automatically indexes.

Consider which span tags are worth creating MetricSets for. Here are some questions you can ask about your environment:

• Are there any attributes I look at when an incident occurs? If you’re running Kubernetes, you could index k8s.pod.name to view the performance of services by specific Kubernetes pods.
• Do I run multiple versions or builds of code at the same time? You could index tags for version or build_id to break down your infrastructure according to specific versions or builds of your applications.
• Do I deploy services in multiple regions or fault domains? It could be useful to view metrics for services by specific region span tags to identify issues with resources in specific regions or zones.

There are two ways to add span tags - instrument your application to create span tags, or add span tags to spans when you send trace data to a Splunk OpenTelemetry Collector.

Instrument your application to create span tags

How you instrument code to create span tags depends on your code’s language. For more information about adding span tags at the instrumentation level, see resources for the language you are instrumenting:

Documentation	Instrumentation SDK
Instrument a Java Application	Splunk distribution of OpenTelemetry Java
Instrument a Node.js Application	SignalFx Tracing Library for JavaScript
Instrument a .NET Application	SignalFx Tracing Library for .NET
Instrument a Python Application	Splunk distribution of OpenTelemetry Python
Instrument a Ruby Application	SignalFx Tracing Library for Ruby
Instrument a PHP Application	SignalFx Tracing Library for PHP

Add span tags with an OpenTelemetry Collector

Include span tags in settings for the batch processor in your OpenTelemetry Collector configuration YAML file. You can create span tags with attributes/newenvironment which adds span tags to any spans that don’t already have the tags, or with attributes/copyfromexistingkey which overrides an existing span tag value.

Establishing business workflows

A business workflow is the start-to-finish journey of the collection of traces associated with a given activity or transaction. Each trace consists of multiple spans, and each span has identifying tags.

As a software engineer, site reliability engineer (SRE), or executive, you can use Business Workflows to monitor and troubleshoot end-to-end transactions in your system. In retail contexts, for example, an end-to-end transaction might encompass initial contact through order fulfillment, as captured by a trace.

You can create rules that correlate traces from a specific service or from multiple services that include the same global span tag. You must be an administrator to configure business workflow rules.

► How to configure business workflow rules

To configure a new rule from Splunk Application Performance Monitoring, follow these steps. There is a difference between enabling a rule and applying it. The enable/disable switch affects an individual rule by turning it on or off. After you modify one or more rules, you then use buttons that act on the entire rule set to save or discard those changes. Changes are not applied unless you save them.

Go to Organization Settings (Found at bottom of Nav Menu) > Business Workflow Configuration. Click New Rule. Select one of the following options from the Rule Type drop-down:

• Global Tag. Define workflows based on the value of a global tag in spans associated with a trace. This correlates traces that contain spans with the global tag.
• Service. Define workflows based on traces that include a service you specify. When a trace matches the rule, you also see a specified tag value or endpoint associated with the trace for the service.

Select a Target Global Tag or Target Service according to the Rule Type you selected.

• Target Global Tag prompts you to select an indexed global tag. When you select a tag, the rule correlates all traces with the global tag. The rule name is based on the global tag you select.
• Target Service prompts you to select a service and specify the Source of Workflow Name, which is extra metadata to view about the workflow. You can select to correlate traces for a service by an endpoint for the initiating span or a span tag value

To save your changes and create the rule, click Create. View the list of rules to confirm the rule you just created is enabled. By default, the newest rule has the highest priority. This means Splunk Application Performance Monitoring applies the new rule before applying any other rules. If there are other rules you want to apply first, adjust the priority of the new rule. To apply the new rule and priority list, click Save Changes.

Read more about configuring business workflow rules here, as well as an example rule configuration. You can also alert on business workflows, a process which is covered in Managing alerts in Splunk APM. 

Implementing use cases

Some high-value use cases you might want to work with are:

• Identify slow database queries using Database Query Performance
• Find performance issues using AlwaysOn Profiling

You can also review other troubleshooting and monitoring use cases here.