Customizing span metadata in Splunk APM
During auto-instrumentation of your services for Splunk Application Performance Monitoring, some metadata related to the framework or library was captured in trace spans. However, you want to customize the captured metadata for your specific use cases.
Solution
The attributes processor is an OpenTelemetry Collector component that modifies metadata of spans, metrics, or logs. Span tags are metadata in the form of key-value pairs that provide information and context about operations they represent. Splunk Application Performance Monitoring recognizes certain tag names and leverages those tags when they are present to augment traces.
The term ”attributes” is used in the context of configuring but the data shows up as “tags” in Splunk Application Performance Monitoring.
By default, the configuration file is in the following folder: /etc/otel/collector/agent_config.yaml. When you edit it, keep in mind the following:
- After span data is ingested into Splunk Observability Cloud, the span and its span tags are not editable.
- The total length of all span tag keys, span tag values, and span annotations cannot exceed 64 kb.
- Span tag keys should be unique within a given span.
- Best practice is to define clear tag names that are used throughout your applications to carry the same information.
There are four main metadata modifications, described in the next sections, that should be configured on the OpenTelemetry Collector. Configuration for each of these always consists of two steps:
- Define the component configuration.
- Enable the component.
Insert metadata definition
Using the attributes processor, you can insert a new attribute (tag) in spans where the key does not already exist. For example, you might want to add an overarching application name. You need to specify:
- Key: what attribute to act upon
- Action: insert
- Value: what value should be associated with the key
Define adding a new tag
processors:
...
attributes/insert:
actions:
- key: "app_name"
value: customer_portal
action: insert
Enable the component you defined
service:
pipelines:
traces:
...
processors:
...
-attributes/insert
Deleting metadata
Using the attributes processor, you can delete an attribute from a span. For example, you might want to remove personally identifiable information (PII). You need to specify:
- Key: what attribute to act upon
- Action: delete
Define deleting a tag
processors:
...
attributes/delete:
actions:
- key: http.user_agent
action: delete
Enable the component you defined
service:
pipelines:
traces:
...
processors:
...
-attributes/delete
Hashing metadata
Using the attributes processor, you can hash (SHA1) an existing attribute value. For example, you might want to hash personal identifiable information (PII). You need to specify:
- Key: what attribute to act upon
- Action: hash
Define hashing a tag
processors:
...
attributes/hash:
actions:
- key: http.user_agent
action: hash
Enable the component you defined
service:
pipelines:
traces:
...
processors:
...
-attributes/hash
Extracting metadata
Using the attributes processor, you can extract values using a regular expression rule from the input key to target keys specified in the rule. For example, you might want to extract information from a URL. You need to specify:
- Key: what attribute to act upon
- Pattern: <regular pattern with named matchers>
- Action: extract
Define extracting information
processors:
...
attributes/extract:
actions:
- key: http.url
pattern: ^(?P<http_protocol>.*):\/\/
action: extract
Enable the component you defined
service:
pipelines:
traces:
...
processors:
...
-attributes/extract
Next steps
If you found this article useful and want to advance your skills, Splunk Education offers a 3-hour, instructor-led course on Configuring Tracing and Profiling for Splunk APM. The hands-on labs in the course will teach you how to:
- Deploy and configure the Splunk OTel Collector
- Auto-instrument applications to send traces
- Configure the OTel Collector to modify trace attributes
- Enable Splunk AlwaysOn Profiling
Click here for the course catalog where you can read the details about this and other Splunk Application Performance Monitoring courses, as well as register.