Skip to main content
Do you build apps on Splunk or are a Splunk admin? If so, we want to hear from you. Help shape the future of Splunk and win a $35 gift card!

Take the Survey
Lantern Home
 
 
Splunk Lantern

Setting up the OpenTelemetry Demo in Kubernetes

  1. Last updated
  2. Save as PDF

 

The OpenTelemetry demo application can be deployed using Kubernetes.

Prerequisites

If you’d like to follow along, you’ll need the following software:

  • git
  • Docker
  • A Kubernetes cluster v1.23+ (you can use minikube if running locally)
  • Helm v3.9
  • Splunk Observability Cloud
  • Splunk Cloud Platform

You’ll want to have at least 4 GB of RAM available on your machine to run the demo application.

Create a new index

You can create a new index for the application logs. This is optional, but for our example, let’s create an index named “astronomyshop”.

  1. In Splunk Cloud Platform, navigate to Settings -> Indexes -> New Index. 
  2. Enter the index properties you want, as shown in the following screenshot. 

clipboard_e8838710b79d27e35e34d7eb187006d3e.png 

For further details on the index creation process, see Create events indexes.

Generate a Splunk HTTP Event Collector token

To use HTTP Event Collector (HEC) to send log data to Splunk Cloud Platform, we need to first generate a token. 

  1. In Splunk Cloud Platform, navigate to Settings -> Data Inputs -> HTTP Event Collector -> New Token.
  2. Provide a name for the token, such as “astronomyshop_token”.
    clipboard_ebee7cbc11cb27d3e8180996885120476.png
  3. Specify which index(es) you’d like the HEC token to send events to. In our case, we’ll send events to the “astronomyshop” index.
    clipboard_e2b1aa18a64140c943603dd7cc305c963.png
  4. After the token has been created successfully, make a note of its value for future reference. 

For further details on the token generation process, see Create an Event Collector token on Splunk Cloud Platform.

Generate a Splunk Observability Cloud access token

While we’ll send log data to Splunk Cloud Platform, we’ll be sending metrics and traces to Splunk Observability Cloud. This requires an access token.

  1. In Splunk Observability Cloud, navigate to the Settings Wheel ☸ -> Access Tokens -> New Token.
  2. Enter a name for the token such as “AstronomyShop Token”.
  3. Because we'll use this token for both ingest and API, select the options shown in the following screenshot.
    clipboard_ed8982073bff09ad39d07b0080edf04f4.png 
  4. Make a note of the token for future reference, as we’ll need it below.  

For further details on the creation process, see Create an access token. The OpenTelemetry demo application generates approximately 5,000 custom metrics, primarily due to metrics related to Kafka and RPC. If you’d like to minimize custom metric usage within your Splunk Observability Cloud account, set a custom metric limit on the token used to instrument the demo application. For more information, see Manage data ingest using access tokens. 

Install the Splunk Distribution of the OpenTelemetry Collector for Kubernetes 

Use the Splunk Observability Cloud guided installation for deploying the Splunk Distribution of the OpenTelemetry Collector for Kubernetes in our Kubernetes cluster. You can find the guidance at Data Management -> + Add Integration -> Kubernetes. Note the following settings:

  • If you’re using minikube, the cluster name is typically “minikube”.
    minikube.png
  • If deploying the demo application to a public cloud environment such as AWS, Azure, or GCP, select the appropriate settings for Provider and Distribution.

Next, we’ll get a list of commands we can use to deploy the Splunk Distribution of the OpenTelemetry Collector to our Kubernetes cluster using Helm.

clipboard_e3e70bb50a7a4c56569edd29cb8f4cef8.png

After these commands are executed, you should start to see data flowing into Splunk Observability Cloud after a minute or so.

clipboard_ea088ef49e787b68fc8d9546a5475c7b7.png


Point to the Splunk Distribution of the OpenTelemetry Collector

Our demo application includes its own OpenTelemetry collector. But for our example, we want to leverage the Splunk Distribution of the OpenTelemetry Collector that’s already running in our Kubernetes cluster, which is already configured to export data to Splunk Observability Cloud.

To do this, we modified the splunk/opentelemetry-demo.yaml and replaced the value of the OTEL_COLLECTOR_ENDPOINT environment variable for each service as follows.

- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: http://$(NODE_IP):4317

This tells each service to utilize the IP address of the node it runs on to connect to the collector.

We also updated the OTEL_RESOURCE_ATTRIBUTES for each service to include the deployment.environment attribute:

- name: OTEL_RESOURCE_ATTRIBUTES
value: service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo,deployment.environment=development

In our example, we set this to “development”, but feel free to change it if you wish.

Install the OpenTelemetry Demo

Navigate to a directory where you’d like to work with the demo application code. For example: 

cd ~/Documents/Code/

Then, grab the source code for the demo repository by running the following commands: 

git clone https://github.com/signalfx/opentelemetry-demo.git

Navigate to this directory: 

cd opentelemetry-demo

Create a namespace for the OpenTelemetry Demo

Next, let’s create a separate namespace to deploy our demo application into.

kubectl create namespace otel-demo

Run the OpenTelemetry Demo

Run the demo application using the following command:

kubectl apply --namespace otel-demo -f 
./splunk/opentelemetry-demo.yaml

After a minute or so you should see the application components appearing on the service map by navigating to APM -> Explore in Splunk Observability Cloud.

clipboard_e2cc0859131a1ce55b2059b161a58a72b.png

Next steps

The OpenTelemetry Demo is running, and is sending metrics and traces to Splunk Observability Cloud. Now, to get logs into Splunk Cloud Platform, let's look at Getting Kubernetes log data Into Splunk Cloud Platform with OpenTelemetry.

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you would like assistance.