Forwarding OCI Streaming to Splunk with Splunk OpenTelemetry Collector for Kafka

A.1 Install dependencies

These packages are used for connectivity testing and producing test messages - not required for the collector itself.

sudo apt-get update 
sudo apt-get install -y kafkacat curl netcat-openbsd 

A.2 Download the collector library

mkdir -p ~/soc4kafka && cd ~/soc4kafka 
wget https://github.com/signalfx/splunk-otel-collector/releases/download/v0.155.0/otelcol_linux_amd64 
chmod +x otelcol_linux_amd64

A.3 Create the secrets file

Create ~/soc4kafka/collector.env and restrict its permissions. This keeps every secret out of the config file and out of process arguments. 

cat > ~/soc4kafka/collector.env <<'EOF' 
KAFKA_BOOTSTRAP=<KAFKA_BOOTSTRAP> 
KAFKA_SASL_USER=<SASL_USERNAME> 
KAFKA_SASL_PASS='<OCI_AUTH_TOKEN>' 
SPLUNK_HEC_URL=<SPLUNK_HEC_ENDPOINT> 
SPLUNK_HEC_TOKEN=<SPLUNK_HEC_TOKEN> 
SPLUNK_INDEX=<SPLUNK_INDEX> 
EOF 
chmod 600 ~/soc4kafka/collector.env

A.4 Create the collector config

Create ~/soc4kafka/config.yaml. Substitute <CONSUMER_GROUP> and <TOPIC> directly in the file; these aren't secrets and don't need to live in the env file. 

receivers: 
  kafka: 
    brokers: 
      - ${env:KAFKA_BOOTSTRAP} 
    group_id: <CONSUMER_GROUP> 
    client_id: <CONSUMER_GROUP> 
    group_rebalance_strategy: range 
    initial_offset: earliest 
    tls: 
      insecure_skip_verify: false 
    auth: 
      sasl: 
        username: ${env:KAFKA_SASL_USER} 
        password: ${env:KAFKA_SASL_PASS} 
        mechanism: PLAIN 
    logs: 
      topics: 
        - <TOPIC> 
      encoding: text 

processors: 
  resourcedetection: 
    detectors: [system] 
    system: 
      hostname_sources: ["os"] 

exporters: 
  splunk_hec: 
    token: ${env:SPLUNK_HEC_TOKEN} 
    endpoint: ${env:SPLUNK_HEC_URL} 
    source: oci-streaming 
    sourcetype: oci:streaming:text 
    index: ${env:SPLUNK_INDEX} 
    tls: 
      insecure_skip_verify: true 
    splunk_app_name: soc4kafka
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      block_on_overflow: true
      sizer: items
      batch:
        min_size: 1000

service: 
  pipelines: 
    logs: 
      receivers: [kafka] 
      processors: [resourcedetection] 
      exporters: [splunk_hec] 

A.5 Verify the connectivity before starting

Check that the VM can reach Splunk HEC:

nc -vz <SPLUNK_HEC_HOST> 8088 

Confirm HEC accepts a posted event: 

set -a; source ~/soc4kafka/collector.env; set +a 
curl -kv \ 
  -H "Authorization: Splunk $SPLUNK_HEC_TOKEN" \ 
  -d '{"event":"hello","index":"'"$SPLUNK_INDEX"'"}' \ 
  "$SPLUNK_HEC_URL/event"

Expect {"text":"Success","code":0}. An Invalid token or Incorrect index error means Splunk setup needs fixing before you go further. 

Check that the VM can reach the Kafka broker and authenticate - the single best end-to-end connectivity test, since success means DNS, routing, TLS, and SASL all work: 

kafkacat -L \ 
  -b "$KAFKA_BOOTSTRAP" \ 
  -X security.protocol=SASL_SSL \ 
  -X sasl.mechanisms=PLAIN \ 
  -X sasl.username="$KAFKA_SASL_USER" \ 
  -X sasl.password="$KAFKA_SASL_PASS" | head -20

You should see <TOPIC> listed. If it times out or returns an auth error, resolve that before proceeding - the collector will exhibit the same failure. 

A.6 Start the collector

Run in the foreground first to watch startup logs:

cd ~/soc4kafka 
set -a; source ./collector.env; set +a 
./otelcol_linux_amd64 --config config.yaml 

A healthy startup looks like: 

Everything is ready. Begin running and processing data. 
...joined, balancing group   group: <CONSUMER_GROUP> 
...synced                    assigned: <TOPIC>[0] 
...beginning heartbeat loop

If you see NOT_COORDINATOR repeating, stop the collector, change group_id and client_id to a new name, and restart. 

A.7 Install as a systemd service

After the collector starts cleanly, promote it to a managed service so it restarts automatically and its logs land in journald.

sudo mkdir -p /opt/soc4kafka /etc/soc4kafka 
sudo cp ~/soc4kafka/otelcol_linux_amd64 /opt/soc4kafka/ 
sudo cp ~/soc4kafka/config.yaml /opt/soc4kafka/ 
sudo cp ~/soc4kafka/collector.env /etc/soc4kafka/collector.env 
sudo chmod 600 /etc/soc4kafka/collector.env 

sudo useradd --system --no-create-home --shell /usr/sbin/nologin otel 
sudo chown otel:otel /opt/soc4kafka/otelcol_linux_amd64 
sudo chown otel:otel /opt/soc4kafka/config.yaml 
sudo chown otel:otel /etc/soc4kafka/collector.env 

sudo tee /etc/systemd/system/soc4kafka.service > /dev/null <<'EOF' 
[Unit] 
Description=SOC4Kafka collector (OCI Streaming -> Splunk) 
After=network-online.target 
Wants=network-online.target 

[Service] 
User=otel 
Group=otel 
EnvironmentFile=/etc/soc4kafka/collector.env 
ExecStart=/opt/soc4kafka/otelcol_linux_amd64 --config /opt/soc4kafka/config.yaml 
Restart=on-failure 
RestartSec=5s 

[Install] 
WantedBy=multi-user.target 
EOF 

sudo systemctl daemon-reload 
sudo systemctl enable --now soc4kafka 
sudo journalctl -u soc4kafka -f

A.8 Send a test message and confirm in the Splunk platform

set -a; source ~/soc4kafka/collector.env; set +a 
printf '{"hello":"splunk","ts":"%s"}\n' "$(date -u +%FT%TZ)" | \ 
  kafkacat -P \ 
    -b "$KAFKA_BOOTSTRAP" \ 
    -t <TOPIC> \ 
    -X security.protocol=SASL_SSL \ 
    -X sasl.mechanisms=PLAIN \ 
    -X sasl.username="$KAFKA_SASL_USER" \ 
    -X sasl.password="$KAFKA_SASL_PASS" 

In Splunk search: index=<SPLUNK_INDEX> sourcetype=oci:streaming:text 

You can also watch collector throughput on the VM - receiver_accepted_log_records_total should increment when you produce, followed shortly by exporter_sent_log_records_total as the batch flushes: 

curl -s http://127.0.0.1:8888/metrics | grep -E 
'otelcol_(receiver_accepted|exporter_sent)' 

B.1 Install MicroK8s

sudo snap install microk8s --classic --channel=1.33/stable 
sudo usermod -a -G microk8s "$USER" 
sudo chown -f -R "$USER" ~/.kube 
newgrp microk8s 

microk8s enable hostpath-storage 
microk8s enable rbac 
microk8s enable metrics-server 

Enable DNS pinned to the OCI VCN resolver, 169.254.169.254 - the same address in every OCI VCN. It answers both private OCI names (your broker's private endpoint) and public names (your Splunk HEC host), so using it as the single upstream matters: 

microk8s enable dns:169.254.169.254 

B.2 Fix the OCI host firewall

The OCI Ubuntu image ships a firewall rule that blocks forwarded traffic, which prevents pods from reaching the Kubernetes API server and causes CoreDNS and Calico to crash-loop (no route to host to the ClusterIP). Remove it: 

sudo iptables -L FORWARD -n --line-numbers | head 
sudo iptables -D FORWARD 1    # removes the REJECT rule (usually at position 1)

Pods recover within about 60 seconds. Make the fix permanent; the rule returns on reboot otherwise:

# Edit the persisted ruleset and remove the REJECT line, then reload: 
sudo grep -nE 'REJECT|icmp-host-prohibited' /etc/iptables/rules.v4 
# Delete the matching line from the file, then: 
sudo netfilter-persistent reload

On a test VM, you can instead disable the OS firewall entirely. The OCI VCN security list still controls ingress at the cloud layer: 

sudo systemctl disable --now netfilter-persistent 

B.3 Create the namespace

microk8s kubectl create namespace soc4kafka 

B.4 Create the Kubernetes secrets

The collector reads credentials from Kubernetes Secrets injected as environment variables; they never appear in the Helm values file. The key names below are required by the chart. Don't rename them. 

# Kafka SASL password — the key name "password" is required by the chart 
microk8s kubectl -n soc4kafka create secret generic kafka-sasl \ 
  --from-literal=password='<OCI_AUTH_TOKEN>' 

# Splunk HEC token — the key name "splunk-hec-token" is required by the chart 
microk8s kubectl -n soc4kafka create secret generic splunk-hec \ 
  --from-literal=splunk-hec-token='<SPLUNK_HEC_TOKEN>'

B.5 Create values.yaml

Create this file on the VM (for example, at ~/soc4kafka_microk8s/values.yaml). Substitute all <PLACEHOLDERS> with your real values. 

replicaCount: 1 

kafkaReceivers: 
  - name: main 
    brokers: 
      - <KAFKA_BOOTSTRAP> 
    client_id: <CONSUMER_GROUP>       # e.g. soc4kafka-m8k-v1 — must be fresh 
    group_id: <CONSUMER_GROUP> 
    group_rebalance_strategy: range 
    initial_offset: earliest 
    logs: 
      topics: 
        - <TOPIC> 
      encoding: text 
    auth: 
      sasl: 
        username: <SASL_USERNAME> 
        mechanism: PLAIN 
        secret: kafka-sasl           # references the Secret created in step B.4 
    tls: 
      insecure_skip_verify: false    # OCI broker cert is publicly trusted 

splunkExporters: 
  - name: primary 
    endpoint: <SPLUNK_HEC_ENDPOINT> 
    secret: splunk-hec               # references the Secret created in step B.4 
    source: oci-streaming 
    sourcetype: oci:streaming:text 
    index: <SPLUNK_INDEX> 
    splunk_app_name: soc4kafka 
    tls: 
      insecure_skip_verify: true    # Splunk default self-signed cert has no SAN 
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000
      block_on_overflow: true
      sizer: items
      batch:
        min_size: 1000

pipelines: 
  - name: oci-to-splunk 
    type: logs 
    receivers: [main] 
    exporters: [primary] 
    processors: [resourcedetection] 

extraEnv: 
  - name: KAFKA_KAFKA_MAIN_SASL_PASSWORD 
    valueFrom: 
      secretKeyRef: 
        name: kafka-sasl 
        key: password 

resources: 
  limits: 
    cpu: 500m 
    memory: 512Mi 
  requests: 
    cpu: 100m 
    memory: 256Mi 

collectorLogs: 
  enabled: false 
collectorMetrics: 
  enabled: false 

B.6 Install the chart

microk8s helm3 repo add splunk-opentelemetry-collector-for-kafka \ 
  https://splunk.github.io/splunk-opentelemetry-collector-for-kafka 
microk8s helm3 repo update 

microk8s helm3 upgrade --install soc4kafka \ 
  splunk-opentelemetry-collector-for-kafka/splunk-opentelemetry-collector-for-kafka \ 
  -n soc4kafka \ 
  -f ~/soc4kafka_microk8s/values.yaml 

B.7 Verify the deployment

Check that all pods are running: 

microk8s kubectl get pods -A 

Tail the collector logs and look for the healthy startup sequence:

microk8s kubectl -n soc4kafka logs -f \ 
  deploy/soc4kafka-splunk-opentelemetry-collector-for-kafka 

Expected output: 

Everything is ready. Begin running and processing data. 
franz   joined, balancing group   group: <CONSUMER_GROUP> 
franz   synced                    assigned: <TOPIC>[0] 
franz   assigning partitions      ... 

If you see NOT_COORDINATOR repeating, change client_id and group_id to a new name in values.yaml and re-run the helm3 upgrade command from step B.6. 

B.8 Send a test message and confirm in the Splunk platform

Produce a message from the VM (install kafkacat first if needed: sudo apt-get install -y kafkacat): 

echo "hello-from-microk8s-$(date -Is)" | kafkacat -P \ 
  -b <KAFKA_BOOTSTRAP> \ 
  -t <TOPIC> \ 
  -X security.protocol=SASL_SSL \ 
  -X sasl.mechanisms=PLAIN \ 
  -X sasl.username='<SASL_USERNAME>' \ 
  -X sasl.password='<OCI_AUTH_TOKEN>' \ 
  -X ssl.ca.location=/etc/ssl/certs/ca-certificates.crt 

Alternatively, use the OCI Console: Streaming → Streams → select stream → Produce Test Message. 

In Splunk search: index=<SPLUNK_INDEX> sourcetype="oci:streaming:text" earliest=-5m