Load balancing traffic to Edge Processors in Amazon EKS

The manifest for creating a LoadBalancer service

Here is a sample manifest for a LoadBalancer service.

apiVersion: v1
kind: Service
metadata:
  name: ep-service
spec:
  type: LoadBalancer
  ports:
    - name: s2s
      protocol: TCP
      port: 9997
      targetPort: 9997
    - name: hec
      protocol: TCP
      port: 8088
      targetPort: 8088
  selector:
    app: ep

While there are many other options available to fine tune and control your data ingress rules, this manifest represents the minimum configuration needed to establish an ingress data path.

This manifest assumes that your Edge Processor will accept data for Splunk clients on the standard port of TCP/9997 and for the HTTP Event Collector on TCP/8088. Because Edge Processor also supports syslog on arbitrary UDP and TCP ports, you might find that you need to add those ports and protocols to the list of ports that are opened.

Additionally, by placing a load balancer between our data and our containers, we have some flexibility in the ports that our data sources can expect to send to and the actual ports that our Edge Processors listen on. This can help address scenarios where the global port settings of your Edge Processors aren’t able to easily be changed, but your data sources require a different port. The most common situation for this to occur is syslog, when your servers are locked to a specific port like 514, but your Edge Processors are listening on a port like 5514 or others.

Finally, our selector needs to match the annotation we provided in our deployment manifest from the previous article (Cloud version / OnPrem version). In your deployments, you will most likely customize these annotations to match and support your overall topology. We’re keeping things simple in this manifest with just ep.

After you’ve applied this manifest, Amazon will provision a classic NLB, and your agents can begin sending data to the NLB name on the appropriate ports and that data will be routed to your Edge Processors. You can find your NLB-friendly name using: kubectl get svc. Alternatively, you can use your AWS console or command line to review the provisioned load balancers.

Autoscaling Edge Processor using Amazon EKS horizontal pod autoscaling

A key benefit of running Edge Processor in Kubernetes is the ability to scale up resources dynamically based on the utilization of the pods using horizontal pod autoscaling (HPA). As with load balancing, the use of EKS offers a lot of out-of-the-box capability without needing to prepare and configure a lot of prerequisites. Before getting into the HPA itself, let’s quickly explore the dimensions we’ll use for scaling.

In many horizontal autoscaling situations, a wide variety of system and application metrics can be measured and used to determine when to scale. The same is true for Edge Processor, but for this example we’ll focus on CPU and memory, which tend to be the best measure for needing additional resources. Over time, as you refine your metrics collection and find trends in your system usage, you might decide that network traffic or other Edge Processor specific metrics are more important, and you can adjust your autoscaling rules.

To get started, we need to install and enable the Kubernetes metrics server. By doing this, we expose memory as a measurable metric since CPU is the only metric exposed in EKS by default. With both CPU and memory available, we can build our HPA.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

kubectl get -n default --raw "/apis/metrics.k8s.io/v1beta1/pods"

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ep-deployment-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ep-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

kubectl get hpa

Next steps

With these steps, you now have a scalable, load-balanced Edge Processor infrastructure in Kubernetes, ready to handle your data routing requirements.