Getting Docker log data into Splunk Cloud Platform with OpenTelemetry

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

In this article, we’ll review a few of the ways we can get logs into Splunk Cloud Platform and demonstrate the recommended way to get Docker logs into Splunk Cloud Platform.

This article uses the Astronomy Shop OpenTelemetry Demo example configured in Setting up the OpenTelemetry Demo in Docker. If you don't have an OpenTelemetry Demo application set up in your environment, use that article first to set one up.

How to use Splunk software for this use case

The Splunk Distribution of the OpenTelemetry Collector includes a number of receivers that we can use to ingest log data:

Filelog receiver. This receiver tails and parses log files.
Fluent Forward receiver. This receiver accepts log events using the Fluent Forward protocol.

We’ll explore both of these options below, starting with File Log Receiver, which is recommended in most environments as it’s more performant and less resource intensive.

Option 1: Use the Filelog receiver

The Filelog Receiver is a popular receiver in the OpenTelemetry community. With this option, we’ll configure the Filelog receiver to read and parse Docker log files. The logs will be processed by the various processors configured in the collector pipeline before being exported to Splunk Cloud Platform via the HEC endpoint.

This option expects Docker log files to be stored in a folder named /var/lib/docker/containers. You’ll need to adjust the configuration if your environment uses a different location for its Docker log files.

Configure the JSON log driver

Let’s start by updating the log driver settings in our docker-compose.yml file to add the container name, image name, and short id to the logs, separated by pipes. We’ll use the pipes in a subsequent step to regex extract the values in the logging pipeline.

x-default-logging: &logging
 driver: "json-file"
 options:
   max-size: "5m"
   max-file: "2"
   tag: "{{.Name}}|{{.ImageName}}|{{.ID}}"

Update the OpenTelemetry Collector service configuration

Next, let’s update the OpenTelemetry Collector service configuration in docker-compose.yml with the following changes:

Ensure that the container runs as “root”, so it’s able to read the /var/lib/docker/containers volume mount from the host machine
Update the command and volumes used to launch the container, to ensure the appropriate collector configuration file is used
Add a read-only volume, so the collector can see the docker logs from the host in the /var/lib/docker/containers directory

The resulting configuration looks like this.

otel-collector:
 image: quay.io/signalfx/splunk-otel-collector:latest
 container_name: otel-collector
 deploy:
   resources:
     limits:
       memory: 200M
 restart: unless-stopped
 command: ["--config=/etc/otelcol-config.yml"]
 user: 0:0
 volumes:
   - ./splunk/otelcol-config-filelog.yml:/etc/otelcol-config.yml
   - ./logs:/logs
   - ./checkpoint:/checkpoint
   - /var/lib/docker/containers:/var/lib/docker/containers:ro
 ports:
   - "${OTEL_COLLECTOR_PORT_GRPC}"
   - "${OTEL_COLLECTOR_PORT_HTTP}"
   - "9464"
   - "8888"
   - "13133"
   - "6060"
   - "9080"
   - "9411"
   - "9943"
 logging: *logging
 environment:
   - ENVOY_PORT
   - HOST_FILESYSTEM
   - OTEL_COLLECTOR_HOST
   - OTEL_COLLECTOR_PORT_GRPC
   - OTEL_COLLECTOR_PORT_HTTP
   - GOMEMLIMIT=160MiB
   - SPLUNK_ACCESS_TOKEN=${SPLUNK_ACCESS_TOKEN}
   - SPLUNK_REALM=${SPLUNK_REALM}
   - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
   - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
   - SPLUNK_MEMORY_TOTAL_MIB=${SPLUNK_MEMORY_TOTAL_MIB}

The collector will utilize the configuration file stored at ./splunk/otelcol-config-filelog.yml

Review the OpenTelemetry Collector configuration

Next, let's review the changes that have already been applied to the OpenTelemetry Collector configuration file (located at ./splunk/otelcol-config-filelog.yml) to add the filelog receiver.

receivers:
 filelog:
   include:
     - /var/lib/docker/containers/*/*-json.log
   encoding: utf-8
   fingerprint_size: 1kb
   force_flush_period: "0"
   include_file_name: false
   include_file_path: true
   max_concurrent_files: 1024
   max_log_size: 1MiB
   operators:
     - id: parser-docker
       timestamp:
         layout: '%Y-%m-%dT%H:%M:%S.%LZ'
         parse_from: attributes.time
       type: json_parser
     - id: extract_metadata_from_docker_tag
       parse_from: attributes.attrs.tag
       regex: ^(?P<name>[^\|]+)\|(?P<image_name>[^\|]+)\|(?P<id>[^$]+)$
       type: regex_parser
       if: 'attributes?.attrs?.tag != nil'
     - from: attributes.name
       to: resource["com.splunk.sourcetype"]
       type: copy
       if: 'attributes?.name != nil'
     - from: attributes.name
       to: resource["docker.container.name"]
       type: move
       if: 'attributes?.name != nil'
     - from: attributes.image_name
       to: resource["docker.image.name"]
       type: move
       if: 'attributes?.image_name != nil'
     - from: attributes.id
       to: resource["docker.container.id"]
       type: move
       if: 'attributes?.id != nil'
     - from: attributes.stream
       to: resource["log.io.stream"]
       type: move
     - field: attributes.attrs.tag
       type: remove
       if: 'attributes?.attrs?.tag != nil'
     - from: attributes.log
       to: body
       type: move
   poll_interval: 200ms
   start_at: beginning

The filelog receiver was also added to the collector pipeline.

pipelines:
 traces:
   receivers: [jaeger, otlp, sapm, zipkin]
   processors: [batch, resourcedetection]
   exporters: [ sapm, signalfx ]
 metrics:
   receivers: [otlp, signalfx, prometheus, hostmetrics]
   processors: [batch, resourcedetection]
   exporters: [signalfx]
 logs:
   receivers: [otlp, signalfx, filelog]
   processors: [batch, resourcedetection]
   exporters: [splunk_hec]

Verify log data is flowing

Restart the demo application by stopping docker compose and then running the following command:

docker compose up --force-recreate --remove-orphans --detach

You should now see log data coming in from all of the services in the application:

We can also run a count by sourcetype, to ensure that we capture logs from all services in the demo application.

Option 2: Using the Fluent Forward receiver

The next option is the Fluent Forward receiver, which leverages Docker’s fluentd logging driver.

Configure the Fluent Forward Receiver

We’ll begin by adding the Fluent Forward receiver to our OpenTelemetry Collector configuration.

These changes have already been applied to ./splunk/otelcol-config-fluentd.yml

Add the following code to the receivers section of the configuration file. This tells the receiver to listen on port 24224 for events using the Fluent Forward protocol.

receivers:
 fluentforward:
   endpoint: 0.0.0.0:24224

We also need to modify the pipelines section of our collector configuration to ensure that the fluentforward receiver is included in the pipeline.

pipelines:
…
 logs:
   receivers: [otlp, signalfx, fluentforward]
   processors: [batch]
   exporters: [splunk_hec]

Configure the Fluentd logging driver

Next, let’s update the docker-compose.yml to instruct Docker to use the fluentd logging driver for our demo application. This is done by replacing the existing logging configuration with the following. This directs the output to our Fluentd instance running on port 24224.

x-default-logging: &logging
 driver: "fluentd"
 options:
   fluentd-address: "localhost:24224"
   fluentd-async: "true"
   fluentd-retry-wait: 1s
   fluentd-max-retries: 10

Update the OpenTelemetry Collector service configuration

Next, let’s update the OpenTelemetry Collector service configuration in docker-compose.yml with the following changes:

Update the command and volumes used to launch the container, to ensure the appropriate collector configuration file is used
Expose the required ports for the collector, including port 24224 for the Fluent Forward receiver

The resulting configuration looks like this.

otel-collector:
 image: quay.io/signalfx/splunk-otel-collector:latest
 container_name: otel-collector
 deploy:
   resources:
     limits:
       memory: 200M
 restart: unless-stopped
 command: ["--config=/etc/otelcol-config.yml"]
 user: 0:0
 volumes:
   - ./splunk/otelcol-config-fluentd.yml:/etc/otelcol-config.yml
   - ./logs:/logs
   - ./checkpoint:/checkpoint
 ports:
   - "${OTEL_COLLECTOR_PORT_GRPC}"
   - "${OTEL_COLLECTOR_PORT_HTTP}"
   - "9464"
   - "8888"
   - "13133"
   - "6060"
   - "9080"
   - "9411"
   - "9943"
   - "24224:24224"   # Fluentforward receiver
 logging: *logging
 environment:
   - ENVOY_PORT
   - HOST_FILESYSTEM
   - OTEL_COLLECTOR_HOST
   - OTEL_COLLECTOR_PORT_GRPC
   - OTEL_COLLECTOR_PORT_HTTP
   - GOMEMLIMIT=160MiB
   - SPLUNK_ACCESS_TOKEN=${SPLUNK_ACCESS_TOKEN}
   - SPLUNK_REALM=${SPLUNK_REALM}
   - SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
   - SPLUNK_HEC_URL=${SPLUNK_HEC_URL}
   - SPLUNK_MEMORY_TOTAL_MIB=${SPLUNK_MEMORY_TOTAL_MIB}

The collector will utilize the configuration file stored at ./splunk/otelcol-config-fluentd.yml

Verify log data is flowing

Restart the application by stopping docker compose and then running the following command.

docker compose up --force-recreate --remove-orphans --detach

You should now see log data coming in from all of the services in the application.

Cleanup

If you want to, you can clean up the application deployment by stopping the docker compose process using Ctrl+C. If you see that containers are still running, you can stop all the containers with the following command:

docker compose stop

Next steps

In this article, we showed how both the Filelog and Fluent Forward receivers can be used to bring Docker logs into Splunk Cloud Platform. You might also be interested in how to do this in Kubernetes. If so, see: Getting Kubernetes log data Into Splunk Cloud Platform with OpenTelemetry.

You might also be interested in configuring Splunk Log Observer Connect to bring the logs into Splunk Observability Cloud, and then using correlated log, trace, and metric data in Splunk Observability Cloud to rapidly troubleshoot application issues.

Finally, the following resources might also help you set up this process from Docker:

Splunk Help: Filelog receiver
Splunk Help: Fluent Forward receiver
Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you would like assistance.