Running Edge Processor nodes in Amazon EKS

Prerequisite: Pushing the Edge Processor node container to a container registry

When you created the Edge Processor node container, it was done on a local server. Because of this, you will need to tag and push the container you created into a registry that is available to your Kubernetes environment. For EKS, a logical location is Amazon ECR, but any similar registry will do. 

As you will see in the Kubernetes deployment manifest section below, we include the container name and the path to that container. Whichever registry you use will be part of the manifest.

Create a configmap for Edge Processor groupId (optional)

As we saw in the prior article, each Edge Processor has a unique groupId that is supplied to the container in order for that node to be properly assigned to the corresponding Edge Processor. This groupId is a GUID and is not particularly reader-friendly when inspecting a deployment manifest. In this example, we will use a Kubernetes configmap to build a friendly name that can be used in the manifest to map to the appropriate GUID.

The syntax for a configmap is straightforward and you can have as many key/value pairs in the data as you have Edge Processors:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ep-instance-guids
data:
  EP_CORP_DC_1: "8de44a16-d638-400d-b796-16abbd59deb7"

Notice that this is just a key/value pair mapping for the friendly name of the Edge Processor and its GUID from the Edge Processor UI as seen here:

Even though creating and maintaining this configmap is a manual process, it’s unlikely that Edge Processors will be frequently added so this should be a relatively low-touch process.

Additionally, be aware that this step is not mandatory. As we will discover shortly, if you prefer, you can directly incorporate the GUID for the Edge Processor into the manifest.  

Using Kubernetes secrets for providing the service principal authentication

As we saw in the prior articles, the container’s entrypoint script calls the “auth” binary to retrieve a current API token during the container start process. We must supply the service principal name and private key to the auth binary. While we could use a configmap or hard-code this, the service principal name and private key are sensitive and should be stored securely. There are many ways to manage secrets, but a straightforward approach is to use native Kubernetes secrets for these items.  

In this deployment example, we’ll use Kubernetes secrets to retrieve the name and key when provisioning the containers and pass them as environment variables to the init script. So we’ll first use the authentication bootstrap process to generate a service principal name and private key and then store them in secrets for later retrieval.  

To create the service principal name secret from the bootstrap process, use:

kubectl create secret generic epsp –from-literal=scssp=<your service principal name> –from-literal=scspk='<your private key>'

In the example above, we’re using kubectl to build the secrets on the command line for brevity. In practice, you might use manifests as we did with the configmap, in which case you will need to base64 encode the service principal values first.

Using a manifest to build your deployment

Download the deployment manifest here. Let’s take a look at the relevant items within the manifest:

Deploying the Edge Processor deployment

Now that we understand the main components of the deployment and have met the requirements, we can apply the manifest and watch the nodes come up using kubectl or other native tools.

kubectl apply -f edgeprocessor.yaml and kubectl get pods
You can also confirm that the Edge Processors have bootstrapped and begun appearing in your Edge Processor user interface.
After a short time, the instances will transition from “Pending” to “Healthy”.

After the nodes have come up and are healthy, you are almost ready to send your data. In situations where the nodes are traditional servers, you could start sending data right away but because we’re in a containerized scenario, we need to expose access to the nodes through a load balancer. We’ll take a look at that in the next article. But before that, let’s briefly explore monitoring these nodes.

Native Kubernetes monitoring in Splunk Observability Cloud

These Edge Processor resources can and should be monitored alongside other Kubernetes resources. This can be accomplished using Splunk Observability Cloud and the capabilities built into that platform for monitoring Kubernetes using the OpenTelemetry agents and instrumentation. This provides node and container level visibility with well-known metrics and topology views into the environment. In addition to infrastructure level monitoring, the output from the Edge Processor runtime can be found in the pod logs as well as in the files located at /splunk-edge/var/log, in particular edge.log. These files can be included in your Kubernetes log monitoring regime. However, as we’ll see next, Edge Processor nodes also send runtime logs and telemetry into your Splunk Enterprise environment, which can be used alongside native Kubernetes monitoring.

Edge Processor metrics and logs in the Splunk platform

Since the Edge Processors are provisioned and managed by Splunk Cloud Platform, the telemetry associated with the Edge Processor nodes (as opposed to the Kubernetes nodes) are automatically collected and sent to your Splunk indexers and can be used for reporting and alerting. All of the Edge Processor logs can be found in your Splunk _internal index using the source type splunk-edge. For example:

index=_internal sourcetype="edge-log"

This search shows the edge.log results for all of your Edge Processor nodes for troubleshooting and creation of searches, dashboards, and alerts.

In addition to the Edge Processor node logs, all of the nodes report a wide variety of performance metrics to the Splunk Cloud Platform _metrics index. You can explore these metrics using the Splunk analytics workbench or through search. You can discover all available metrics reported by Edge Processor using the following search:

| mcatalog values(metric_name) WHERE index=_metrics AND sourcetype="edge-metrics"

While we don’t explore the metrics and logs in this article, it’s worth exploring that data and considering your scale strategy based on the data provided.