Skip to main content
Lantern Home

 

Splunk Lantern

Running Edge Processor nodes in Amazon EKS (OnPrem)

  1. Last updated
  2. Save as PDF

 

Now that you have Edge Processor nodes running from containers, the next step in scaling Edge Processors is to to deploy, scale, and manage those containers with Kubernetes. In this series, we’re using Amazon EKS as our platform because it’s very common and a lot of the heavy Kubernetes lifting has been done by Amazon, in particular autoscaling, metrics, and load balancing. The details in this and the following article are generally compatible with other distributions of Kubernetes, but the specific implementation might vary slightly depending on the platform.

Prerequisite: Pushing the Edge Processor node container to a container registry

When you created the Edge Processor container, it was done on a local server. Because of this, you will need to tag and push the container you created into a registry that is available to your Kubernetes environment. For EKS, a logical location is Amazon ECR, but any compatible registry will work. The steps for pushing container images to a registry are out of scope for this series.

As you will see in the Kubernetes deployment manifest section below, we include the container name and the path to that container. Whichever registry you use will be part of the manifest.

Determine Edge Processor GroupID Management

As we saw in previous articles, each Edge Processor has a unique groupId that is supplied to the container in order for that instance to be properly assigned to the corresponding Edge Processor. This groupId is a GUID and is not particularly reader-friendly when inspecting a deployment manifest, but the entrypoint script requires that a groupId GUID be provided. There are several possible approaches for adding it:

  • Copy/paste: The groupId is in the UI for each Edge Processor. This ID can be copied from the UI and pasted directly into the manifest as the GROUP_ID environment variable. The manifest will require comments or manual inspection to understand which deployment belongs to which Edge Processor.
    1.png 
    …manifest…
- name: GROUP_ID
  value: d81d22ae-6913-4ccf-a7d1-4afa6c7081bd
…manifest…
  • Use a ConfigMap: The use of a ConfigMap allows for human readable manifests by pairing the name of the Edge Processor with its groupId. A ConfigMap like this can be crafted manually or built via API:
    • Build a ConfigMap using the Splunk API. This approach requires a valid authentication token. It can be the same token as we have previously created. The following command fetches the list of Edge Processors, uses jq to make key value pairs from the results, and then uses kubectl to create a ConfigMap called “ep-guids”. Here is an example: 
      curl --location 'https://<dm_control_plane>:8089/servicesNS/-/splunk_pipeline_builders/edge/v1alpha3/processors' --header 'Authorization: Bearer <your_token>' | jq -r '.[] | "\(.name)=\(.id)"' | xargs -I {} kubectl create configmap ep-guids --from-literal={}
    • Use the ConfigMap in the manifest. Now we can use the human readable form of the Edge Processor, and the value passed to the container will be the GUID value from the ConfigMap. 
      …manifest…
- name: GROUP_ID 
  valueFrom: 
  configMapKeyRef: 
  name: ep-guids 
  key: EP_CORP_DC_1 
…manifest…

Use Kubernetes secrets for token authentication

As we saw in the previous article covering authentication requirements, the container’s entrypoint script expects a token.

In this article’s deployment example, we’ll use Kubernetes secrets to store and retrieve the token. To create the secret:

  1. Create/retrieve a token using one of the methods described in the previous article.
  2. Create a secret using that token: kubectl create secret generic EdgeProcessorSecrets –from-literal=token=<token from step 1>

In the example above, we use kubectl to build the secret on the command line for brevity. In practice, you might use manifests or an API as we did with the ConfigMap.

Edge Processor deployment manifest

Download the deployment manifest here. Let’s take a look at the relevant items within the manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ep-deployment
spec:
  replicas: 1 # specify the number of replicas
  selector:
    matchLabels:
      app: ep
  template:
    metadata:
      labels:
        app: ep

Boilerplate deployment configuration. Customize to fit your environment as it relates to any annotations, labels, selectors, etc.

The replicas setting dictates the number of instances for this Edge Processor configuration.
spec:
  containers:
  - name: ep
    image: 123456.dkr.ecr.us-east-2.amazonaws.com/ep

Update the image location to point to your container registry and the name of the container for EP that you have built.
env:
  - name: DMX_TOKEN #private key
    valueFrom:
      secretKeyRef:
        name: EdgeProcessorSecrets
        key: token		 Set the token environment variable from the secrets we set above. This is required for the bootstrap process.
 - name: GROUP_ID
   valueFrom:
    configMapKeyRef:
       name: ep-instance-guids
       key: EP_CORP_DC_1		 Set or retrieve the groupId GUID for the Edge Processor being configured. This example uses a human readable name “EP_CORP_DC_1” via a configmap. This could also be a hard-coded GUID.
  - name: DMX_HOST
    value: "<DM Control Plane IP address or FQDN>"
resources:
  requests:
    cpu: "2000m"
    memory: "2Gi"

Some final settings:

  • Your control plane hostname or IP address.
  • Default resources assigned to each pod. There are many different approaches to resource allocation in Kubernetes and are not specific to Edge Processor.

In your environment, you might choose to name your deployment something else, include additional annotations, set resource requests and limits, or put the deployment into a specific namespace. For purposes of this example, we’ve kept the deployment as straightforward as possible.

Apply the Edge Processor deployment

Now that we understand the main components of the deployment and have met the requirements, we can apply the manifest and watch the nodes come up using kubectl or other native tools.

Deploy from the command line

Run the following:

kubectl apply -f edgeprocessor.yaml
kubectl get pods

Monitor from the UI

New instances will appear pending They will transition to healthy
3.png 4.png

After the nodes have come up and are healthy, you are almost ready to send your data. In situations where the nodes are traditional servers, you could start sending data right away but because we’re in a containerized scenario, we need to expose access to the nodes through a load balancer. We’ll take a look at that in the next article. But before that, let’s briefly explore monitoring these nodes.

Monitor the Edge Processor deployment

As with all infrastructure, it’s important to monitor and measure your Edge Processors to ensure uninterrupted and reliable data flow. Additionally, while we’ll explore horizontal pod autoscaling in the next article, we can use the results from monitoring to decide when to scale up or down our Edge Processor deployment replica count. In this Kubernetes scenario, there are two distinct and complementary approaches to monitoring: monitoring natively in Splunk Observability Cloud, or monitoring metrics and logs in the Splunk platform.

Native Kubernetes monitoring in Splunk Observability Cloud

These Edge Processor resources can and should be monitored alongside other Kubernetes resources. This can be accomplished using Splunk Observability Cloud and the capabilities built into that platform for monitoring Kubernetes using the OpenTelemetry agents and instrumentation. This provides node and container level visibility with well-known metrics and topology views into the environment. 

In addition to infrastructure level monitoring, the output from the Edge Processor runtime can be found in the pod logs as well as in the files located at /splunk-edge/var/log, in particular edge.log. These files can be included in your Kubernetes log monitoring regime.

However, as we’ll see next, Edge Processor nodes also send runtime logs and telemetry into your Splunk Enterprise environment, which can be used alongside native Kubernetes monitoring.

Edge Processor metrics and logs in the Splunk platform

Edge Processors automatically send extensive telemetry associated with the Edge Processor runtime (as opposed to the Kubernetes pods) to the control plane instance. If your control plane instance is forwarding logs to your main indexers, you will be able to use normal searches and analytics dashboards to measure and monitor.

All the Edge Processor logs can be found in your Splunk _internal index using the source type splunk-edge. For example:

index=_internal sourcetype="edge-log"

This search shows the edge.log results for all of your Edge Processor nodes for troubleshooting and creation of searches, dashboards, and alerts.

In addition to the Edge Processor node logs, all of the nodes report a wide variety of performance metrics to the _metrics index. You can explore these metrics using the Splunk analytics workbench or through search. You can discover all available metrics reported by Edge Processor using the following search:

| mcatalog values(metric_name) WHERE index=_metrics AND sourcetype="edge-metrics"

While we don’t explore the metrics and logs in this article, it’s worth exploring that data and considering your scale strategy based on the data provided.

Next steps

Now that we’ve got our Edge Processor deployment up and running and scaled for our needs, we need to create a path from our data sources into our Edge Processor nodes that are running in containers. To learn how, read the next article in this series, Load balancing traffic to Edge Processors in Amazon EKS.