Skip to main content
Os artigos do Splunk Lantern estão agora disponíveis em português.
Splunk Lantern

Running Edge Processor nodes in Amazon EKS


Now that we have Edge Processor nodes running from containers, we can move on to deploying, scaling, and managing those containers with Kubernetes. In this series, we’re using Amazon EKS as our platform because it’s very common and a lot of the heavy Kubernetes lifting has been done by Amazon, in particular autoscaling, metrics, and load balancing. The details in this and the following article are generally compatible with other distributions of Kubernetes, but the specific implementation might vary slightly depending on the platform.

Using your Edge Processor node container in EKS

Prerequisite: Pushing the Edge Processor node container to a container registry

When you created the Edge Processor node container, it was done on a local server. Because of this, you will need to tag and push the container you created into a registry that is available to your Kubernetes environment. For EKS, a logical location is Amazon ECR, but any similar registry will do. 

The steps for pushing container images to a registry are out of scope for this series.

As you will see in the Kubernetes deployment manifest section below, we include the container name and the path to that container. Whichever registry you use will be part of the manifest.

Create a configmap for Edge Processor groupId (optional)

As we saw in the prior article, each Edge Processor has a unique groupId that is supplied to the container in order for that node to be properly assigned to the corresponding Edge Processor. This groupId is a GUID and is not particularly reader-friendly when inspecting a deployment manifest. In this example, we will use a Kubernetes configmap to build a friendly name that can be used in the manifest to map to the appropriate GUID.

The syntax for a configmap is straightforward and you can have as many key/value pairs in the data as you have Edge Processors:

apiVersion: v1
kind: ConfigMap
  name: ep-instance-guids
  EP_CORP_DC_1: "8de44a16-d638-400d-b796-16abbd59deb7"

Notice that this is just a key/value pair mapping for the friendly name of the Edge Processor and its GUID from the Edge Processor UI as seen here:


Even though creating and maintaining this configmap is a manual process, it’s unlikely that Edge Processors will be frequently added so this should be a relatively low-touch process.

Additionally, be aware that this step is not mandatory. As we will discover shortly, if you prefer, you can directly incorporate the GUID for the Edge Processor into the manifest.  

Using Kubernetes secrets for providing the service principal authentication

As we saw in the prior articles, the container’s entrypoint script calls the “auth” binary to retrieve a current API token during the container start process. We must supply the service principal name and private key to the auth binary. While we could use a configmap or hard-code this, the service principal name and private key are sensitive and should be stored securely. There are many ways to manage secrets, but a straightforward approach is to use native Kubernetes secrets for these items.  

In this deployment example, we’ll use Kubernetes secrets to retrieve the name and key when provisioning the containers and pass them as environment variables to the init script. So we’ll first use the authentication bootstrap process to generate a service principal name and private key and then store them in secrets for later retrieval.  

To create the service principal name secret from the bootstrap process, use:

kubectl create secret generic epsp –from-literal=scssp=<your service principal name> –from-literal=scspk='<your private key>'

The private key must be wrapped in single quotes to preserve the JSON formatting.

In the example above, we’re using kubectl to build the secrets on the command line for brevity. In practice, you might use manifests as we did with the configmap, in which case you will need to base64 encode the service principal values first.

Using a manifest to build your deployment

Download the deployment manifest here. Let’s take a look at the relevant items within the manifest:

kind: Deployment By using a deployment, we can set the number of replicas and rely on Kubernetes to maintain that number of instances.
image: <your registry and image name>

As indicated above, we need to host our container in a registry that is accessible by our Kubernetes environment. If you are using ECR, this would look like:

- name: SCS_PK
      name: epsp
      key: scspk
- name: SCS_SP
      name: epsp
      key: scssp
For both the private key and the service principal name, we set the appropriate environment variables to the secret keys that we configured above.
 - name: GROUP_ID
       name: ep-instance-guids
       key: EP_CORP_DC_1
We can (optionally) use the configmap we created above to use a friendly name in the manifest to retrieve the proper groupId for our Edge Processor. In this example, the GUID associated with EP_CORP_DC_1 will be placed in GROUP_ID.
- name: SCS_TENANT
  value: "<your-tenant>"
- name: SCS_ENV
  value: "production"
The remaining environment variables are set to static values to satisfy the initscript.

In your environment, you might choose to name your deployment something else, include additional annotations, set resource requests and limits, or put the deployment into a specific namespace. For purposes of this example, we’ve kept the deployment as straightforward as possible.

Deploying the Edge Processor deployment

Now that we understand the main components of the deployment and have met the requirements, we can apply the manifest and watch the nodes come up using kubectl or other native tools.

kubectl apply -f edgeprocessor.yaml


kubectl get pods

You can also confirm that the Edge Processors have bootstrapped and begun appearing in your Edge Processor user interface. 3.png
After a short time, the instances will transition from “Pending” to “Healthy”. 4.png

After the nodes have come up and are healthy, you are almost ready to send your data. In situations where the nodes are traditional servers, you could start sending data right away but because we’re in a containerized scenario, we need to expose access to the nodes through a load balancer. We’ll take a look at that in the next article. But before that, let’s briefly explore monitoring these nodes.

Monitoring the Edge Processor deployment

As with all infrastructure, it’s important to monitor and measure your Edge Processors to ensure uninterrupted and reliable data flow. Additionally, while we’ll explore horizontal pod autoscaling in the next article, we can use the results from monitoring to decide when to scale up or down our Edge Processor deployment replica count. In this Kubernetes scenario there are two distinct and complementary approaches to monitoring.

Native Kubernetes monitoring in Splunk Observability Cloud

These Edge Processor resources can and should be monitored alongside other Kubernetes resources. This can be accomplished using Splunk Observability Cloud and the capabilities built into that platform for monitoring Kubernetes using the OpenTelemetry agents and instrumentation. This provides node and container level visibility with well-known metrics and topology views into the environment. In addition to infrastructure level monitoring, the output from the Edge Processor runtime can be found in the pod logs as well as in the files located at /splunk-edge/var/log, in particular edge.log. These files can be included in your Kubernetes log monitoring regime. However, as we’ll see next, Edge Processor nodes also send runtime logs and telemetry into your Splunk Enterprise environment, which can be used alongside native Kubernetes monitoring.

Edge Processor metrics and logs in the Splunk platform

Since the Edge Processors are provisioned and managed by Splunk Cloud Platform, the telemetry associated with the Edge Processor nodes (as opposed to the Kubernetes nodes) are automatically collected and sent to your Splunk indexers and can be used for reporting and alerting. All of the Edge Processor logs can be found in your Splunk _internal index using the source type splunk-edge. For example:

index=_internal sourcetype="edge-log"

This search shows the edge.log results for all of your Edge Processor nodes for troubleshooting and creation of searches, dashboards, and alerts.

In addition to the Edge Processor node logs, all of the nodes report a wide variety of performance metrics to the Splunk Cloud Platform _metrics index. You can explore these metrics using the Splunk analytics workbench or through search. You can discover all available metrics reported by Edge Processor using the following search:

| mcatalog values(metric_name) WHERE index=_metrics AND sourcetype="edge-metrics"

While we don’t explore the metrics and logs in this article, it’s worth exploring that data and considering your scale strategy based on the data provided.

Next steps

Now that we’ve got our Edge Processor deployment up and running and scaled for our needs, we need to create a path from our data sources into our Edge Processor nodes that are running in containers. To learn how, read the next article in this series, Load balancing traffic to Edge Processors in Amazon EKS.