Skip to main content
 
 
 
Splunk Lantern

Customizing span metadata in Splunk APM

 

During auto-instrumentation of your services for Splunk Application Performance Monitoring, some metadata related to the framework or library was captured in trace spans. However, you want to customize the captured metadata for your specific use cases.

Solution 

The attributes processor is an OpenTelemetry Collector component that modifies metadata of spans, metrics, or logs. Span tags are metadata in the form of key-value pairs that provide information and context about operations they represent. Splunk Application Performance Monitoring recognizes certain tag names and leverages those tags when they are present to augment traces. 

The term ”attributes” is used in the context of configuring but the data shows up as “tags” in Splunk Application Performance Monitoring.

By default, the configuration file is in the following folder: /etc/otel/collector/agent_config.yaml. When you edit it, keep in mind the following: 

  • After span data is ingested into Splunk Observability Cloud, the span and its span tags are not editable.
  • The total length of all span tag keys, span tag values, and span annotations cannot exceed 64 kb.
  • Span tag keys should be unique within a given span.
  • Best practice is to define clear tag names that are used throughout your applications to carry the same information.

There are four main metadata modifications, described in the next sections, that should be configured on the OpenTelemetry Collector. Configuration for each of these always consists of two steps:

  1. Define the component configuration.
  2. Enable the component.

Insert metadata definition

Using the attributes processor, you can insert a new attribute (tag) in spans where the key does not already exist. For example, you might want to add an overarching application name. You need to specify:

  • Key: what attribute to act upon
  • Action: insert
  • Value: what value should be associated with the key

Define adding a new tag

processors:

... 

  attributes/insert:
    actions:
      - key: "app_name"
        value: customer_portal
        action: insert

Enable the component you defined


service:

  pipelines:

    traces:

      ...

      processors:

        ...

        -attributes/insert

Deleting metadata

Using the attributes processor, you can delete an attribute from a span. For example, you might want to remove personally identifiable information (PII). You need to specify:

  • Key: what attribute to act upon
  • Action: delete

Define deleting a tag

processors:

... 

  attributes/delete:
    actions:
      - key: http.user_agent
        action: delete

Enable the component you defined


service:

  pipelines:

    traces:

      ...

      processors:

        ...

        -attributes/delete

Hashing metadata

Using the attributes processor, you can hash (SHA1) an existing attribute value. For example, you might want to hash personal identifiable information (PII). You need to specify:

  • Key: what attribute to act upon
  • Action: hash

Define hashing a tag

processors:

... 

  attributes/hash:
    actions:
      - key: http.user_agent
        action: hash

Enable the component you defined


service:

  pipelines:

    traces:

      ...

      processors:

        ...

        -attributes/hash

Extracting metadata

Using the attributes processor, you can extract values using a regular expression rule from the input key to target keys specified in the rule. For example, you might want to extract information from a URL. You need to specify:

  • Key: what attribute to act upon
  • Pattern: <regular pattern with named matchers>
  • Action: extract

Define extracting information

processors:

... 

  attributes/extract:
    actions:
      - key: http.url
        pattern: ^(?P<http_protocol>.*):\/\/
        action: extract

Enable the component you defined


service:

  pipelines:

    traces:

      ...

      processors:

        ...

        -attributes/extract

Next steps

If you found this article useful and want to advance your skills, Splunk Education offers a 3-hour, instructor-led course on Configuring Tracing and Profiling for Splunk APM. The hands-on labs in the course will teach you how to:

  • Deploy and configure the Splunk OTel Collector
  • Auto-instrument applications to send traces
  • Configure the OTel Collector to modify trace attributes
  • Enable Splunk AlwaysOn Profiling

Click here for the course catalog where you can read the details about this and other Splunk Application Performance Monitoring courses, as well as register.