Splunk Operator for Kubernetes: Advanced operational learnings

Symptoms

The symptoms observed for this problem included:

• High system CPU, often between 55% and 99.9% system CPU on the Linux server.
• iostat would show 100% (or even 100.10%) disk utilization on multiple disks, but the disk writes across all involved NVMe drives (mdraid with RAID level 0) would show 0 writes during the issue.
• Attempting to launch a shell inside the indexer pod would freeze (blank screen) until the issue subsided.
• Zombie processes would increase at the host level during the issue and disappear when the issue subsided.
• Searches would continue to involve the problematic indexer, as the indexer was still online but responding extremely slowly.
• At the Splunk level, the search head experienced long handoff times when initiating searches and took noticeably longer to return results. From the user’s perspective, dashboard panels appeared to be nearly loaded but remained in that state for minutes rather than seconds. This behavior led to search head clusters hitting concurrency limits, resulting in skipped searches — and a frustrated user base.

Restarting the Splunk indexer would fully resolve the issue, provided the indexer eventually responded to the offline or stop request, which could itself take several minutes.

The elevated system CPU usage typically persisted for about 10 minutes before often subsiding. However, as the workload increased, the issue began to occur for longer durations.

To measure the issue in real-time I used top and iostat. I also tried iotop, Redhat eBPF tools (bcc-tools), and considered tools such as glances or htop, but these tools did not reveal any new information. While a bcc-tool like ext4slower could identify slow I/O transactions, all I/O was slow during the problem.

Our first attempt to solve this issue

After removing swap from the hosts “just in case”, we attempted to mitigate the issue by enabling the Slow peer disconnect feature in Splunk. Within the limits.conf file the slow_peer_disconnect stanza mentions:

# This stanza contains settings for the heuristic that will detect and
# disconnect slow peers towards the end of a search that has returned a
# large volume of data.

This solution does not apply to the "handoff" time, which involves the indexer in the initial phase of a search. Unfortunately, even though we accepted that we wouldn’t have perfectly accurate search results in a failure scenario, this option did not help.

Tracing the performance issue

You can use the limits.conf to enable search_metrics:

[search_metrics]
debug_metrics = true

This setting can be dynamically enabled or disabled by updating the limits.conf file on the filesystem, requiring no reload or restart.

debug_metrics shows statistics on a per-indexer basis, including the “handoff” time, which is normally a single value across all indexers. However if you have a large number of peers, the job inspector becomes unreadable due to the large number of responses.

In the Alerts for Splunk Admins application (Splunkbase or GitHub) the below search provides the handoff time per-indexer:

SearchHeadLevel - Job performance data per indexer handoff time

Or for more detailed information, you can use:

SearchHeadLevel - Job performance data per indexer

These searches use the jobs endpoint to help view the performance per-indexer, the former only works with debug_metrics enabled.

Clarifying the issue

I eventually identified the issue as being related to memory usage within the cgroup or container. We have memory limits on all indexer pods, for example, 150GB on the larger indexers.

When the memory limit is exceeded, the OOM killer terminates the process. However, if the pod approaches the limit, I suspect the Linux system initiates an aggressive memory reclamation resulting in degraded performance.

This behavior is similar to how a Linux system thrashes its swap file when memory is scarce. The key difference here is that there is no swap file within the pod, and swap on the host was proactively disabled as a preventative measure.

At the OS level, this behavior is largely hidden, with the only observable behaviors including elevated system CPU usage and the pod becoming almost unresponsive. It’s only by inspecting the cgroup that you can observe the cache availability dropping and other activity consistent with a low-memory condition in Linux.

The issue isn’t exclusive to Kubernetes, as I’ve encountered similar behavior on Linux servers utilizing swap. Based on my research, there’s little documentation addressing performance degradation within pods or Linux cgroups under memory stress. While I’ve made some assumptions about the underlying cause, the implemented solution seems effective, which leads me to believe memory pressure is the root cause.

Solution discussion

My environment uses cgroupsv1. Although a memory pressure file exists within the cgroup, it can’t be read directly, and you must subscribe to it via the cgroup notification API.

This GitHub gist provides an example C implementation to subscribe to memory pressure events. A C example for a cgroup event listener also exists, and there is also a python equivalent.

Since memory pressure appeared to be challenging to measure, I found that cgroups have a memory.limit_in_bytes and a memory.usage_in_bytes. While this provides a simple calculation, it is not useful as Linux servers can cache memory.

Fortunately, the cgroups directory contains a memory.stat file with numerous metrics in plain text. A few relevant metrics included:

cache 110069547008
rss 42799886336
pgpgin 192985903033
pgpgout 192948581417
pgfault 205169464127
pgmajfault 1545206

Since the page faults, page-in, and page-out metrics seemed to be cumulative counters, I opted to use the available cache value instead, as it offered a more straightforward approach.

cgroupsv2 seems to provide even more statistics, which might simplify this process.

Solution

This bash script was developed to address the issue and is configured for use with containerd. If you’re using a different container runtime, you might need to replace nerdctl with ctr or an equivalent command.

The code below:

• Looks for any Splunk containers.
• For each Splunk container, it identifies its cgroup directory.
• If the memory.stat file shows the cache as below the threshold (800MB in the below code), it then finds the largest search process. If there is enough memory, the loop continues to the next container or exits.
• After logging the details of the largest search process, it sends a kill signal to that process.
• After a 2-second sleep, the memory check re-runs and it kills a process if required or exits the loop.

The while loop is included because I found that when an indexer pod nears its memory limit, terminating the largest search alone often isn’t sufficient for recovery. The loop ensures the pod’s memory usage drops back below the threshold.

In my environment, this script is configured to run every 20 seconds to prevent issues:

#!/bin/bash

# Log file
LOG_FILE="/opt/splunkforwarder/var/log/splunk/splunk_oom_killer.log"
MAX_LOG_SIZE=$((2 * 1024 * 1024))  # 2MB in bytes
# MB at which point we kill searches due to a potential issue
low_threshold=800

# Logging function
log() {
    local message="$1"
    if [ -f "$LOG_FILE" ]; then
        local log_size
        log_size=$(stat -c%s "$LOG_FILE")
        if [ "$log_size" -ge "$MAX_LOG_SIZE" ]; then
            mv "$LOG_FILE" "${LOG_FILE}.1"
        fi
    fi
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $message" | tee -a "$LOG_FILE"
}

cache_check() {
    local dir="$1"
    # use cache as the proxy measure for memory that can be freed if required
    # memory.limit_in_bytes - memory.usage_in_bytes did not prove to be a good measure
    # memory.pressure_level will work well but you must be "notified" of the issue, you cannot just read it as a text file
    # for now keeping this simple and just watching the cache
    cache=`grep "^cache" $dir/memory.stat | awk '{ print $2 }'`
    megabytes=$(awk "BEGIN {printf \"%.0f\", $cache / (1024 * 1024)}")
    log "Memory in cache: $megabytes MB for $id and instance $instance"
    # if we have less than $low_threshold MB remaining, prepare for process killing
    if [ $megabytes -lt ${low_threshold} ]; then
      log "Container id=$id, instance $instance is below the available cache of ${low_threshold} currently with memory=$megabytes MB available"
      pid=`ps -eo pid,rss,command --sort=-rss --cols 1400 | grep -E "search-launcher|search --id" | grep -Ff $dir/cgroup.procs | head -n 1 | awk '{ print $1 }'`
      if [ "x$pid" = "x" ]; then
        log "Unable to find a pid to kill -- $pid"
        return 0
      fi      
      log "Killing $pid -- `ps -o user,pid,%cpu,%mem,vsz,rss,tty,stat,start,time,args -p $pid --cols 1400`"
      kill $pid
      return 1
    fi
    return 0
}

#ctr shows all containers, nerdctl is showing running containers
#list=`ctr -n k8s.io containers list | grep "splunk:" | awk '{ print $1 }'`
nerdctl -n k8s.io ps --no-trunc | grep "splunk:" | awk '{ print $1 " " $NF }' > /tmp/oom_killer_pod_ids.txt
while read -r id instance; do
  # find any Splunk pods and check memory limits
  dir=`find /sys/fs/cgroup/memory -name "*$id"`
  if [ "x$dir" = "x" ]; then
      log "Unable to find directory for $id, continue"
      continue
  fi
  while true; do
      cache_check "$dir"
      if [ $? -eq 0 ]; then
        break  # Exit the loop if cache_check succeeds
      fi
      # give the system time to update the stats
      sleep 2
  done
done < /tmp/oom_killer_pod_ids.txt

chown splunk:splunk $LOG_FILE

Since deploying this solution, indexer overload caused by memory exhaustion has stopped completely.

SmartStore downloads still present challenges. Unfortunately, a solution for this is not currently available beyond increasing cache sizes or reducing workload.

To further prevent memory issues, the built-in memory tracker in limits.conf can also be utilized:

[search]
enable_memory_tracker = true
search_process_memory_usage_threshold = 50000

Data model acceleration jobs can consume up to 50GB of memory, so this setting is configured high in our environment. Often, a lower value can be used (though a rolling restart is required).

Reducing the chance of re-occurrence

Splunk platform version 9.3 introduced a new feature called usage-based data rebalancing that helps distribute search workloads more evenly among peers.

To help with running data rebalancing more frequently, I’ve built an application, Automatic Data Rebalance (Splunkbase or GitHub), that provides a modular input to trigger indexer cluster data rebalancing on a schedule. This modular input only triggers a rebalance when when the search factor is met and the cluster is not in maintenance mode.

Search head clusters

Since publishing the previous article, we’ve begun running search head clusters within the Kubernetes environment. Initially, there was an issue where the deployer required resource parity with all search head cluster members, but this has been resolved in SOK version 2.7.1.

Generally, search heads worked seamlessly within Kubernetes.

Platform upgrades

Upgrading Splunk platform with the SOK has proved quite straightforward, as the SOK appears to be following the recommended upgrade order.

Since the searchable rolling restart of the indexer cluster temporarily suspends data model acceleration searches, we were unable to perform a fully automated upgrade. The idea Splunk indexing tier searchable rolling restart should allow the scheduler to run jobs as expected, which is expected to be implemented in a future version of the platform, could allow searchable rolling restarts and upgrades on the indexing tier.

Indexer scaling

Given our existing investment in the Kubernetes stack, we’re now procuring higher-capacity hardware. This results in larger cost savings, as a server with double the CPU, memory, and disk capacity does not cost twice as much. This also helps reduce our data center footprint.

The upgraded high-capacity hardware also provides increased flexibility for mixing workloads. Indexers with greater CPU requirements can temporarily utilize the unused CPU resources of those handling fewer active searches.

Disk cache limits

We discovered that pods ingesting approximately 200GB of data per day were unable to sustain heavy search workloads with 7TB disks (providing 5.5TB of usable cache), as the capacity proved insufficient.

Unfortunately, this led to SmartStore downloads, which temporarily disrupted ingestion and degraded search performance. To address this, we plan to invest in higher-capacity disks for the servers and reduce ingestion workloads on those that cannot be upgraded.

Server restarts

In the previous article, I noted that server-level restarts did not gracefully shut down the Splunk platform, and this issue persists. However, I’ve made several updates to my systemd unit file and associated scripts to better manage search head restarts. For the Splunk pods, implementing a PreStop hook could be an effective solution, as this would ensure that any shutdown event automatically triggers a splunk offline or splunk stop command.

Backups

Velero provides Kubernetes configuration backups. I’ve tested a full restore in development and the only issue I had was duplicated replicaSets, otherwise this tool works perfectly.