Customizing the Splunk OpenTelemetry distribution to accommodate unsupported use cases

Google Cloud

The following steps cover Google Cloud setup. Only run these commands within the context of a non-production project that is safe to use for experimentation. It is highly recommended that you create a dedicated Google Cloud project for the purposes of this exercise. For example, to create a new project titled "oteltest" and switch configuration contexts to this new project, issue the following commands:

$ gcloud projects create oteltest

$ gcloud config set project oteltest

1. Export variables. Create or identify an existing Google Cloud project which can be used for this experiment. Export its name into an environment variable. Ensure this variable is set to the projectId as listed in the output from gcloud projects list --format=flattened. For example:

   $ gcloud projects list --format=flattened
   ---
   createTime:     2022-06-30T21:56:08.101Z
   lifecycleState: ACTIVE
   name:           oteltest
   parent.id:      <REDACTED>
   parent.type:    organization
   projectId:      oteltest
   projectNumber:  <REDACTED>
   ---
   ...
   

   $ export GOOGLE_CLOUD_PROJECT=oteltest

   Note that projectId is not always the same as name.

   Next, export the following shell variables:

   $ export SERVICE_ACCOUNT_SHORT=otel-test
   $ export SERVICE_ACCOUNT_FULL=${SERVICE_ACCOUNT_SHORT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
   $ export SERVICE_ACCOUNT_FILENAME=gcp.json
   $ export SINK_NAME=otel-log-sink
   $ export PUBSUB_TOPIC=otel-log-topic
   $ export PUBSUB_SUB=otel-log-sub
   

2. Create a service account:

   $ gcloud iam service-accounts create ${SERVICE_ACCOUNT_SHORT}
   

3. Export the JSON key and download it to your development environment:

   $ gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_FILENAME} \
   --iam-account=${SERVICE_ACCOUNT_FULL}
   

4. Enable the Pub/Sub API:

   $ gcloud services enable pubsub.googleapis.com
   

5. Create the Pub/Sub topic:

   $ gcloud pubsub topics create ${PUBSUB_TOPIC}
   

6. Create the Pub/Sub subscription:

   $ gcloud pubsub subscriptions create \
     --topic ${PUBSUB_TOPIC} ${PUBSUB_SUB}
   

7. Bind the role to the service account: 

   $ gcloud pubsub subscriptions add-iam-policy-binding ${PUBSUB_SUB} \
   --member="serviceAccount:${SERVICE_ACCOUNT_FULL}" \
   --role="roles/pubsub.subscriber"
   

8. Create a log sink. The following command creates a log router that matches basic cloud audit logs along with a custom log:

   $ gcloud logging sinks create ${SINK_NAME} \
   pubsub.googleapis.com/projects/${GOOGLE_CLOUD_PROJECT}/topics/${PUBSUB_TOPIC} \
   --log-filter='log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/policy") OR log_id("oteltest")'
   

9. Allow sink writer permission to topic. First, determine the identity of the log sink writer:

   $ export SINK_SERVICE_ACCOUNT=`gcloud logging sinks describe ${SINK_NAME} --format="value(writerIdentity)"`
   

   Next, grant the log writer service account identity permission to publish to the topic:

   $ gcloud pubsub topics add-iam-policy-binding ${PUBSUB_TOPIC} \
    --member="${SINK_SERVICE_ACCOUNT}" --role="roles/pubsub.publisher"
   

Splunk platform

The following steps create a Splunk platform index and HEC token. You must have administrative permissions to perform these operations. The index name and HEC token will be referenced in later steps.

Export environment variables

Since your configuration file references multiple environment variables, you need to declare them in your shell environment before launching the agent.

These environment variables are described below:

• SPLUNK_HEC_TOKEN - Splunk HEC token
• SPLUNK_HEC_URL - URL path to the Splunk HEC endpoint, such as https://mysplunkhec.com:8088/
• SPLUNK_INDEX - Splunk index name to store log messages
• GOOGLE_APPLICATION_CREDENTIALS - Path to GCP service account JSON credential
• GOOGLE_CLOUD_PROJECT - GCP project name
• PUBSUB_SUB - GCP Pub/Sub subscription name

Export each of them before launching the OpenTelemetry agent, as shown below:

$ export SPLUNK_HEC_URL="https://mysplunkhec.com:8088"
$ export SPLUNK_HEC_TOKEN="your-hec-token"
$ export SPLUNK_INDEX="oteltest"
$ export GOOGLE_APPLICATION_CREDENTIALS="/Users/mhite/repo/blog/custom-agent/splunk-otel-collector/gcp.json"

The GOOGLE_CLOUD_PROJECT and PUBSUB_SUB environment variables should already be present from previous setup steps.

Launch the agent

Launch your newly compiled agent, making sure to point to the correct platform binary, using the following command:

$ ./bin/otelcol_darwin_arm64 --config=./pubsub-otel.yaml

The output should look like this:

2023/05/15 15:39:15 settings.go:332: Set config to [./pubsub-otel.yaml]
2023/05/15 15:39:15 settings.go:385: Set ballast to 168 MiB
2023/05/15 15:39:15 settings.go:401: Set memory limit to 460 MiB
2023-05-15T15:39:15.632-0700    info    service/telemetry.go:113    Setting up own telemetry...
2023-05-15T15:39:15.632-0700    info    service/telemetry.go:136    Serving Prometheus metrics    {"address": ":8888", "level": "Basic"}
2023-05-15T15:39:15.632-0700    info    processor/processor.go:300    Development component. May change in the future.    {"kind": "processor", "name": "logstransform", "pipeline": "logs"}
2023-05-15T15:39:15.632-0700    info    service/service.go:141    Starting otelcol...    {"Version": "v0.76.1-50-gb57c7279", "NumCPU": 10}
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:41    Starting extensions...
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:44    Extension is starting...    {"kind": "extension", "name": "pprof"}
2023-05-15T15:39:15.632-0700    info    pprofextension@v0.77.0/pprofextension.go:71    Starting net/http/pprof server    {"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":":1888"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:48    Extension started.    {"kind": "extension", "name": "pprof"}
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:44    Extension is starting...    {"kind": "extension", "name": "zpages"}
2023-05-15T15:39:15.632-0700    info    zpagesextension@v0.77.0/zpagesextension.go:64    Registered zPages span processor on tracer provider    {"kind": "extension", "name": "zpages"}
2023-05-15T15:39:15.632-0700    info    zpagesextension@v0.77.0/zpagesextension.go:74    Registered Host's zPages    {"kind": "extension", "name": "zpages"}
2023-05-15T15:39:15.632-0700    info    zpagesextension@v0.77.0/zpagesextension.go:86    Starting zPages extension    {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:48    Extension started.    {"kind": "extension", "name": "zpages"}
2023-05-15T15:39:15.632-0700    info    extensions/extensions.go:44    Extension is starting...    {"kind": "extension", "name": "health_check"}
2023-05-15T15:39:15.632-0700    info    healthcheckextension@v0.77.0/healthcheckextension.go:45    Starting health_check extension    {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2023-05-15T15:39:15.633-0700    warn    internal/warning.go:51    Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-05-15T15:39:15.633-0700    info    extensions/extensions.go:48    Extension started.    {"kind": "extension", "name": "health_check"}
2023-05-15T15:39:15.633-0700    info    internal/resourcedetection.go:136    began detecting resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "logs"}
2023-05-15T15:39:15.645-0700    info    internal/resourcedetection.go:150    detected resource information    {"kind": "processor", "name": "resourcedetection", "pipeline": "logs", "resource": {"host.id":"BCB8A4F5-B5B2-5132-AB53-1320A7635FCE","host.name":"mattirl-en0.int.beatmixed.net","os.type":"darwin"}}
2023-05-15T15:39:15.730-0700    info    healthcheck/handler.go:129    Health Check state change    {"kind": "extension", "name": "health_check", "status": "ready"}
2023-05-15T15:39:15.730-0700    info    service/service.go:158    Everything is ready. Begin running and processing data.
2023-05-15T15:39:15.730-0700    info    internal/handler.go:117    Starting Streaming Pull    {"kind": "receiver", "name": "googlecloudpubsub", "data_type": "logs"}

Generate and verify delivery of a log message

To test the pipeline, write a message to Cloud Logging and search for it in the Splunk platform. To inject a test message, run the following command:

$ gcloud logging write oteltest "Test message"

Then, within the Splunk platform, search for the following:

index="oteltest" data.textPayload="Test message"

Ensure that you have selected an appropriately constrained time selection within the time picker.

Google Cloud

To avoid future charges, delete the following cloud resources created during this experiment:

• Log sink:

  $ gcloud logging sinks delete ${SINK_NAME}
  

• Pub/Sub subscription:

  $ gcloud pubsub subscriptions delete ${PUBSUB_SUB}
  

• Pub/Sub topic:

  $ gcloud pubsub topics delete ${PUBSUB_TOPIC}
  

• Service account:

  $ gcloud iam service-accounts delete ${SERVICE_ACCOUNT_FULL}
  

Splunk platform

Delete the following Splunk platform resources created during this experiment: