Skip to main content
Lantern Home

 

Splunk Lantern

Export Google Cloud or Google Trusted Partner Cloud (TPC) logs to Splunk Enterprise using the Splunk OpenTelemetry agent

  1. Last updated
  2. Save as PDF

The Splunk OpenTelemetry Collector is a distribution of the OpenTelemetry Collector that provides a unified way for you to receive, process, and export metric, trace, and log data to Splunk Enterprise, Splunk Cloud Platform, and Splunk Observability Cloud products. Using the included Google Cloud Pub/Sub receiver and the Splunk HTTP Endpoint Collector (HEC) exporter, you can export logs from Google Cloud and Trusted Partner Cloud environments into a Splunk Enterprise index.

How to use Splunk software for this use case

This article shows an example of how this use case can be accomplished by configuring the Splunk OpenTelemetry collector to transport Google Cloud log messages from a Pub/Sub subscription to a Splunk HTTP Endpoint Collector (HEC). You'll follow these steps:

  1. Set up
  2. Install and configure the collector
  3. Start the collector
  4. Test
  5. Clean up

This is an experimental approach and might not be production ready. You will not receive support from Splunk or Google Cloud. For issues with the collector, please engage with the OpenTelemetry contrib community in the GitHub repository. For information on deploying a production-ready Google Cloud log export to the Splunk platform, refer to the official Google Cloud documentation.

Prerequisites

  • Familiarity with using the command-line interface (CLI)
  • A Splunk instance with Splunk Add-on for Google Cloud Platform installed
  • Ubuntu-based VM running in Google Cloud to host the OpenTelemetry agent
  • Splunk OpenTelemetry Collector version 0.141.0 or higher

Set up

Google Cloud

The following steps cover Google Cloud setup. Only run these commands within the context of a non-production project that is safe to use for experimentation. It is highly recommended that you create a dedicated Google Cloud project for the purposes of this exercise. For example, to create a new project titled "oteltest" and switch configuration contexts to this new project, issue the following commands:

$ gcloud projects create oteltest
$ gcloud config set project oteltest
  1. Export variables. Create or identify an existing Google Cloud project which can be used for this experiment. Export its name into an environment variable. Ensure this variable is set to the projectId as listed in the output from gcloud projects list --format=flattened. For example:
    $ gcloud projects list --format=flattened
---
createTime:     2022-06-30T21:56:08.101Z
lifecycleState: ACTIVE
name:           oteltest
parent.id:      <REDACTED>
parent.type:    organization
projectId:      oteltest
projectNumber:  <REDACTED>
---
...
$ export GOOGLE_CLOUD_PROJECT=oteltest

    Note that projectId is not always the same as name.

  2. Export the following shell variables: 
    $ export SERVICE_ACCOUNT_SHORT=otel-test
$ export SERVICE_ACCOUNT_FULL=${SERVICE_ACCOUNT_SHORT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
$ export SINK_NAME=otel-log-sink
$ export PUBSUB_TOPIC=otel-log-topic
$ export PUBSUB_SUB=otel-log-sub
  3. Create a service account for the OpenTelemetry agent virtual machine:
    $ gcloud iam service-accounts create ${SERVICE_ACCOUNT_SHORT}
  4. Enable the Pub/Sub API:
    $ gcloud services enable pubsub.googleapis.com
  5. Create the Pub/Sub topic:
    $ gcloud pubsub topics create ${PUBSUB_TOPIC}
  6. Create the Pub/Sub subscription:
    $ gcloud pubsub subscriptions create \
  --topic ${PUBSUB_TOPIC} ${PUBSUB_SUB}
  7. Bind the required roles to the service account:
    $ gcloud pubsub subscriptions add-iam-policy-binding ${PUBSUB_SUB} \
  --member="serviceAccount:${SERVICE_ACCOUNT_FULL}" \
  --role="roles/pubsub.subscriber"
    $ gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} \
  --member="serviceAccount:${SERVICE_ACCOUNT_FULL}" \
  --role="roles/logging.logWriter"
    $ gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} \
  --member="serviceAccount:${SERVICE_ACCOUNT_FULL}" \
  --role="roles/monitoring.metricWriter"
  8. Create a log sink. The following command creates a log router that matches basic cloud audit logs along with a custom log:
    $ gcloud logging sinks create ${SINK_NAME} \
pubsub.googleapis.com/projects/${GOOGLE_CLOUD_PROJECT}/topics/${PUBSUB_TOPIC} \
  --log-filter='log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/policy") OR log_id("oteltest")'
  9. Allow sink writer permission to topic. First, determine the identity of the log sink writer:
    $ export SINK_SERVICE_ACCOUNT=`gcloud logging sinks describe ${SINK_NAME} --format="value(writerIdentity)"`
  10. Next, grant the log writer service account identity permission to publish to the topic: 
    $ gcloud pubsub topics add-iam-policy-binding ${PUBSUB_TOPIC} \
 --member="${SINK_SERVICE_ACCOUNT}" --role="roles/pubsub.publisher"

Splunk platform

The following steps create a Splunk platform index and HEC token. You must have administrative permissions to perform these operations. The index name and HEC token will be referenced in later steps.

Create the index

  1. In the top right corner, click the Settings menu.
  2. Under the Data column, click Indexes.
  3. In the top right corner, click New Index.
  4. In the Index Name field, enter "oteltest".
  5. Click Save.
    new-index (1).png

Create HEC token

  1. In the top right corner, click the Settings menu.
  2. Under the Data column, click Data inputs.
  3. In the HTTP Event Collector row, click Add new under the Actions column.
  4. In the Name field, type "otelhec".
  5. In the top right corner, click Next.
  6. In the Select Allowed Indexes section, choose the oteltest item in the Available item(s) box. You should see oteltest appear in the Selected item(s) box. You should also see the Default Index update to oteltest.

hec-input-settings (1).png

  1. In the top right corner, click Review.
  2. In the top right corner, click Submit.
  3. Copy the Token Value to a text file for later use.

Install and configure the collector

On an Ubuntu virtual machine running in Google Cloud or a TPC, install and configure the Splunk OpenTelemetry collector. When creating the virtual machine, attach the service account created in the Google Cloud setup section.

Install the collector

echo "Setting up Splunk Otel repo..."
curl -sSL https://splunk.jfrog.io/splunk/otel-...k-B3CD4420.gpg > /etc/apt/trusted.gpg.d/splunk.gpg
echo 'deb https://splunk.jfrog.io/splunk/otel-collector-deb release main' > /etc/apt/sources.list.d/splunk-otel-collector.list

# Packages
echo "Updating packages..."
apt-get -y update

echo "Installing packages..."
apt-get -y install splunk-otel-collector

Configure the collector

Create the /etc/otel/collector/splunk-otel-collector.conf file with the following configuration. Replace the items enclosed in <> with values specific to your environment. 

# Required

OTELCOL_OPTIONS=""
SPLUNK_CONFIG=/etc/otel/collector/agent_config.yaml

# The following environment variables are referenced in the /etc/otel/collector/agent_config.yaml config file.

SPLUNK_HEC_URL=https://<IP_ADDRESS_OF_SPLUNK_HEC>:8088/services/collector
SPLUNK_HEC_TOKEN=<SPLUNK_HEC_TOKEN>
SPLUNK_MEMORY_TOTAL_MIB=1024
SPLUNK_LISTEN_INTERFACE=127.0.0.1
GOOGLE_CLOUD_PROJECT=<GOOGLE_CLOUD_PROJECT_NAME>
PUBSUB_SUB=otel-log-sub
SPLUNK_INDEX=oteltest
# only if using a custom TPC endpoint, otherwise comment out or omit
# different TPCs have different endpoints
GOOGLE_PUBSUB_ENDPOINT=pubsub.apis-berlin-build0.goog:443

Create the /etc/otel/collector/agent_config.yaml file with the following configuration. Values enclosed in ${} are automatically substituted with values from the splunk-otel-collector.conf defined above. 

processors:
  batch:
    # gather messages into 10s batches
    timeout: 10s
  resourcedetection:
    # this will ensure the "host" field of our hec message is populated
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  logstransform:
    operators:
      # this parses the payload as JSON and puts it in internal key "attributes"
      # everything in attributes ends up as a HEC "fields" key/value
      - type: json_parser
        parse_from: body
        # grab timestamp
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # now that we've parsed the payload as JSON, blow away current internal key "body"
      - type: remove
        field: body
      # hacky way to encapsulate parsed JSON inside "data" so it works with the GCP-TA
      - type: add
        field: body.data
        value: EXPR(attributes)
      # let's blow away attributes so we don't send any "fields" to HEC
      - type: remove
        field: attributes

extensions:
  health_check:
      endpoint: 0.0.0.0:13133
  pprof:
      endpoint: :1888
  zpages:
  text_encoding:

receivers:
  googlecloudpubsub:
    project: "${GOOGLE_CLOUD_PROJECT}"
    subscription: "projects/${GOOGLE_CLOUD_PROJECT}/subscriptions/${PUBSUB_SUB}"
    encoding: text_encoding
    endpoint: "${GOOGLE_PUBSUB_ENDPOINT}"

exporters:
  splunk_hec:
    token: "${SPLUNK_HEC_TOKEN}"
    endpoint: "${SPLUNK_HEC_URL}"
    source: "otel"
    index: "${SPLUNK_INDEX}"
    sourcetype: "google:gcp:pubsub:message"
    disable_compression: false
    timeout: 10s
    tls:
      insecure_skip_verify: true
  debug:
    verbosity: detailed

service:
  extensions: [ pprof, zpages, health_check, text_encoding ]
  pipelines:
    logs:
      receivers: [ googlecloudpubsub ]
      processors: [ batch, resourcedetection, logstransform ]
      #exporters: [ debug ]
      exporters: [ splunk_hec ]

Start the collector

systemctl stop splunk-otel-collector
systemctl start splunk-otel-collector

# Inspect the status and logs.

systemctl status splunk-otel-collector
journalctl -u splunk-otel-collector

Test

You are now ready to test and verify. You can issue these commands from the OpenTelemetry agent VM.

To test the pipeline, write a message to Cloud Logging and search for it in the Splunk platform. To inject a test message, run the following command:

$ gcloud logging write oteltest "Test message"

Then, within the Splunk platform, search for the following:

index="oteltest" data.textPayload="Test message"

Ensure that you have selected an appropriately constrained time selection within the time picker.

Clean up

Follow these cleanup steps for both the Google Cloud and Splunk platform environments.

Google Cloud

Remove the virtual machine used for hosting the OpenTelemetry agent. Also, to avoid future charges, delete the following cloud resources created during this experiment:

  • Log sink: $ gcloud logging sinks delete ${SINK_NAME}
  • Pub/Sub subscription: $ gcloud pubsub subscriptions delete ${PUBSUB_SUB}
  • Pub/Sub topic: $ gcloud pubsub topics delete ${PUBSUB_TOPIC}
  • Service account: $ gcloud iam service-accounts delete ${SERVICE_ACCOUNT_FULL}

Splunk platform

Delete the following Splunk platform resources created during this experiment:

Delete HEC token

  1. In the top right corner, click the Settings menu.
  2. Under the Data column, click Data inputs.
  3. Click the HTTP Event Collector hyperlink.
  4. In the filter box, type "otelhec" and click enter.
  5. Under the Actions column for the row labeled otelhec, click Delete.
  6. In the Delete Token popup, click Delete.

Delete index

  1. In the top right corner, click the Settings menu.
  2. Under the Data column, click Indexes.
  3. In the filter box, type "oteltest" and click enter.
  4. Under the Actions column for the row labeled oteltest, click Delete.
  5. In the Delete Index popup, click Delete.

You've now seen the basics of configuring the Splunk OpenTelemetry distribution to handle a Google Cloud log pipeline.

Next steps

The following resources might help you understand and implement the guidance in this article: