Skip to main content
Lantern Home

 

Splunk Lantern

Using ingest actions to route data to Google Cloud Storage

  1. Last updated
  2. Save as PDF

This article walks through the configuration steps required to set up ingest actions to route data to Google Cloud Storage using the S3-compatible XML API endpoint.

The configuration discussed in this guide is compatible with a self-managed Splunk Enterprise deployment or a heavy forwarder. It is not available for Splunk Cloud Platform. Additionally, it is an experimental approach and might not be production ready. You will not receive support from Splunk or Google Cloud.

Prerequisites

  • Familiarity with using the command-line interface (CLI)
  • Splunk Enterprise instance (self-hosted)
  • Administrative access to a Google Cloud project

Google Cloud configuration

  1. Create a bucket. Ingest actions will need a bucket to write objects. Use the following command to create a bucket. 
    gcloud storage buckets create gs://<BUCKET_NAME>
    You should never use an existing bucket with ingest actions.
  2. Create a service account. 
    gcloud iam service-accounts create <SERVICE_ACCOUNT_ID>
  3. Create an HMAC key for service account. Ingest actions uses an HMAC key associated with this service account to perform write operations on the bucket. 
    gcloud storage hmac create <SERVICE_ACCOUNT_EMAIL>
    This command outputs the HMAC access ID and secret now associated with the specified service account. Store these securely for use in ingest actions setup steps. The secret cannot be retrieved from Google later.
  4. Grant IAM role to the service account. Ingest actions needs permission to write objects to the storage bucket. Use the following IAM grant command to provide roles/storage.objectCreator on the bucket to the service account. 
    gcloud storage buckets add-iam-policy-binding gs://<BUCKET_NAME> \
    --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
    --role="roles/storage.objectCreator"

Configure the ingest actions destination in Splunk Enterprise 

  1. Within Splunk Enterprise, navigate to Settings > Ingest actions.
  2. Select Destinations > New Destination > S3.
  3. Fill in step one of the S3 setup form with the following fields:
    Field Value
    Destination Title gcs
    S3 bucket name YOUR BUCKET NAME
    S3 endpoint https://storage.googleapis.com
  4. Select Next.
  5. Fill in step two of the S3 setup form with the following fields:
    Field Value
    Access Key ID ACCESS KEY ID FROM HMAC
    Secret Access Key ACCESS KEY FROM HMAC
  6. Select Test Connection. The test should succeed.
  7. Click Save. The Google Cloud Storage bucket is now available as a destination in your ingest actions routing rules.

Additional resources

These resources might help you understand and implement this guidance: