Managing apps with the Splunk Operator for Kubernetes App Framework

AppRepo and remote volumes

The App Framework is configured per CR using the appRepo structure, which defines the polling interval, default scope/volume, app sources, and backing storage volumes.

apiVersion: enterprise.splunk.com/v4 
kind: ClusterManager 
metadata: 
  name: cm 
  finalizers: 
  - enterprise.splunk.com/delete-pvc 
spec: 
  appRepo: 
    appsRepoPollIntervalSeconds: 300 
    defaults: 
      volumeName: app_repo 
      scope: cluster 
    appSources: 
      - name: core 
        location: core/ 
      - name: cm  
        location: cm/ 
        scope: local 
      - name: idxc  
        location: idxc/ 
      - name: tas  
        location: tas/ 
      - name: indexes       
        location: indexes/ 
    volumes: 
      - name: app_repo 
        storageType: s3 
        provider: aws 
        path: <s3 bucket>/<environment>/<deployment>/ 
        endpoint: https://s3-eu-west-2.amazonaws.com 
        region: eu-west-2 
        secretRef: s3-secret

Key concepts: 

• volumes: describe remote storage (type, provider, endpoint, path, region, credentials). 
• appSources: logical groupings of apps under a location within a volume, with a scope and implied install path. 
• defaults: provide default scope and volumeName for appSources that do not override them. 
• appsRepoPollIntervalSeconds: controls how frequently SOK polls remote storage for new/changed app archives

Scope semantics 

The scope determines where an app is installed and how it propagates: 

• local: app is installed locally on the pod(s) governed by the CR (Standalone, LicenseManager, MonitoringConsole, Cluster Manager itself, or Deployer in a SearchHeadCluster CR). 
• cluster: app is staged on the configuration manager (Cluster Manager or Deployer) and then distributed to peers or search heads using native Splunk clustering mechanisms. 

Scope support by CR type: 

Change detection and update mechanics

The App Framework calculates and stores a checksum for each app archive based on the remote object’s Etag and compares it with the previously recorded value in the CR status. When a difference is detected during polling (or a manual trigger), the operator downloads the updated package, stages it, and installs or pushes the new version to the appropriate Splunk components. 

Key behaviors: 

• Archive filenames must remain constant. The framework relies on the Etag of a remote object at a stable path, so updating an app requires overwriting the object with the same filename. 
• Only .spl and .tgz archives are considered; other files are ignored. 
• Deletion is not automated. Removing a package from remote storage does not trigger app removal; disabling must be handled via app.conf in a new archive or by manual cleanup. 
• For clustered deployments, bundle application (apply cluster-bundle) is coordinated by the operator, with SPLUNK_SKIP_CLUSTER_BUNDLE_PUSH defaulting to true so that bundle pushes are initiated by the App Framework logic itself. 

Core apps (platform baseline) 

Core apps are mandatory apps and add‑ons that form the platform baseline and must be present on all relevant tiers (indexer, search head, Monitoring Console (MC), and License Manager (LM)) in every environment (dev, test, prod, etc.). Examples include Splunk Supporting Add‑on for Active Directory, core CIM add‑ons, and internal support apps for monitoring, RBAC, and logging. 

Characteristics: 

• Deployed broadly ("everywhere relevant") with tightly controlled versioning. 
• Changes are infrequent and generally tied to Splunk or premium app upgrades. 
• Owned and governed centrally, with strong regression testing requirements. 

Configuration‑only apps 

Configuration‑only apps consist primarily of .conf files used to define indexes, authentication configuration, roles, macros, limits, and similar non‑visual settings. They should not contain dashboards or search content meant for end users. 

Typical examples: 

• Index definition apps deployed via Cluster Manager to indexer peers (cluster scope). 
• Search‑time configuration used by multiple functional apps, such as macro libraries or lookup definitions. 
• Environment‑specific configuration overlays (for example, dev/test/prod index naming or retention differences). 

Technology add‑ons (TAs) 

TAs are focused on data ingestion, sourcetype definitions, props/transforms, and modular inputs. The App Framework is particularly suited to centralizing and standardizing TA deployment for sources onboarded across multiple clusters. 

Characteristics: 

• Deployed on input tier (heavy forwarder (HF) or standalone CR acting as a HF), indexers (for index‑time parsing where necessary), and search heads (for knowledge objects), depending on Splunk best practices for that TA. 
• Often maintained at different version cadences than user‑facing apps, especially when tied to third‑party product versions. 

User‑facing apps 

User‑facing apps provide dashboards, saved searches, alerts, and reports used directly by analysts or business users. These are typically more numerous, change more frequently, and might have different lifecycles per team or domain. 

Characteristics: 

• Deployed mainly to SearchHeadCluster CRs (cluster scope) or standalone SHs (local scope). 
• Often built and maintained by application teams rather than central Splunk platform teams. 
• Might need environment‑specific branches or feature flags. 

Cluster Manager (indexer cluster management) 

Cluster Manager CR is responsible for distributing cluster‑wide apps to indexer peers and hosting local admin apps. 

Recommended grouping: 

spec: 
  appRepo: 
    defaults: 
      volumeName: app_repo 
      scope: cluster 
    appSources: 
      - name: core          # baseline apps 
        location: core/ 
      - name: cm            # cluster manager local apps 
        location: cm/ 
        scope: local 
      - name: idxc          # core search apps, CIM, etc. 
        location: idxc/ 
      - name: tas           # technical add-ons 
        location: tas/ 
      - name: indexes       # index configuration 
        location: indexes/ 

Usage: 

• Core apps, index definitions, and indexer‑side TAs are deployed with cluster scope from the Cluster Manager to peers. 
• Local admin and tooling apps remain scoped local and run only on the Cluster Manager node. 

Search Head Cluster and Deployer 

SearchHeadCluster CR handles both cluster‑wide search head apps and Deployer‑local admin apps. 

Recommended grouping: 

spec: 
  appRepo: 
    defaults: 
      volumeName: app_repo 
      scope: cluster 
    appSources: 
      - name: core          # baseline apps 
        location: core/ 
      - name: shcd          # deployer local apps 
        location: shcd/ 
        scope: local 
      - name: shcd          # core search apps, CIM, etc. 
        location: shcd/ 
      - name: tas           # technical add-ons 
        location: tas/ 
      - name: user          # user-facing apps and dashboards 
        location: user/ 
      - name: indexes       # index configuration 
        location: indexes/

Usage: 

• Core and TA apps are standardized across search heads via cluster scope. 
• User apps are packaged and versioned independently, but still delivered cluster‑wide to keep the user experience consistent across members. 
• Local Deployer tools (for example, operational dashboards for bundle status) are in shcd. 

Standalone, MonitoringConsole, LicenseManager 

These CRs support only local scope, which simplifies the design. 

spec: 
  appRepo: 
    defaults: 
      volumeName: app_repo 
      scope: local 
    appSources: 
      - name: stdln/mc/lm   # dependent on role 
        location: stdln/mc/lm

Usage patterns: 

• For HFs or standalone SHs acting as data ingress points, TAs and config‑only apps are deployed locally. 
• MonitoringConsole and LicenseManager generally receive a minimal set of core/config apps and any vendor‑provided monitoring packs. 

Recommended layout 

Use a hierarchy that separates by environment, deployment, and role, while keeping filenames consistent for versioning. 

Example S3 layout:

s3://<bucketname>/<environment>/<deployment>/ 
├── core/       	# Apps shared across all resources 
│   ├── app-1/ 
│   └── app-n/ 
├── stdln/  	       # Apps for stand-alone deployments 
│   └── app3/ 
├── idxc/           # Apps only for indexer clusters 
│   ├── app-1/ 
│   └── app-n/ 
├── shc/            # Apps only for search heads 
│   ├── app-1/ 
│   └── app-n/ 
├── cm/             # Apps only for cluster manager 
│   ├── app-1/ 
│   └── app-n/ 
├── shd/            # Apps only for search head cluster deployer 
│   ├── app-1/ 
│   └── app-n/ 
├── lm/             # Apps only for license manager 
│   ├── app-1/ 
│   └── app-n/ 
├── mc/             # Apps only for monitoring console 
│   ├── app-1/ 
│   └── app-n/ 
├── user/  		# User apps may be mapped to multiple resource types 
│   ├── app-1/ 
│   └── app-n/ 
├── tas/  		# TAs may be mapped to multiple resource types 
│   ├── ta-1/ 
│   └── ta-n/ 
├── indexes/        # Apps for index configuration 
│   ├── app-1/ 
│   └── app-n/ 
... 
 
This pattern is repeated for all deployments in environments – (test, uat, prod, etc.)   

This layout allows each environment’s CRs to point to distinct path values under the same or different buckets (for example, path: sok-apps/prod/deployment-1/idxc/). It cleanly separates app versions by environment while reusing the same internal grouping (core/config/tas/user). 

File naming and versioning 

Given that the App Framework requires stable filenames to detect updates, version management should be handled inside the archive rather than by renaming the file. 

Recommendations: 

• Keep filenames stable per logical app (for example, TA_network.tgz). 
• Track app versions in 
  • app.conf (version and build attributes). 
  • Your CI/CD system, tagging builds and releases. 
• To roll back, upload a previous version of the archive with the same name; the App Framework will detect the new Etag and redeploy. 

For human readability and traceability, include the version in a metadata file in the bucket or maintain a separate manifest under source control rather than encoding it in the archive filename. 

Separate buckets or paths per environment 

Each environment (dev, test, prod) uses a distinct path and potentially a separate bucket or storage account. 

Pros: 

• Strong separation of environments and blast radius. 
• Simple promotion flow: copy archive from dev path to test and prod paths when validated. 
• If using separate buckets can have separation of responsibilities 

Cons: 

• Requires additional storage management scripts or CI/CD jobs to copy between paths or buckets. 

Separate paths per environment 

Example S3 layout and CR snippet:

s3://sok-apps/<environment>/<deployment>/ 
├── core/       	 
│   ├── app-1/ 
│   └── app-n/ 
├── stdln/        
│   └── app-1/ 
│   └── app-n/ 
├── idxc/         
│   ├── app-1/ 
│   └── app-n/ 
├── shc/            
│   ├── app-1/ 
│   └── app-n/ 
... 
 
# dev namespace 
spec: 
  appRepo: 
    volumes: 
      - name: app_repo 
        storageType: s3 
        provider: aws 
        path: sok-apps/<environment>/<deployment>/ 
        ... 
 
# prod namespace 
spec: 
  appRepo: 
    volumes: 
      - name: app_repo 
        storageType: s3 
        provider: aws 
        path: sok-apps/<environment>/<deployment>/ 
        ... 

Separate buckets per environment 

Example S3 layout and CR snippet: 

s3://sok-apps-<environment>/<environment>/<deployment>/ 
├── core/       	 
│   ├── app-1/ 
│   └── app-n/ 
├── stdln/        
│   └── app-1/ 
│   └── app-n/ 
├── idxc/         
│   ├── app-1/ 
│   └── app-n/ 
├── shc/            
│   ├── app-1/ 
│   └── app-n/ 
... 
 
# dev namespace 
spec: 
  appRepo: 
    volumes: 
      - name: app_repo 
        storageType: s3 
        provider: aws 
        path: sok-apps-<environment>/<environment>/<deployment>/ 
        ... 
 
# prod namespace 
spec: 
  appRepo: 
    volumes: 
      - name: app_repo 
        storageType: s3 
        provider: aws 
        path: sok-apps-prod/deployment-1/ 

Git‑backed source of truth and CI/CD 

While SOK pulls from object storage, the authoritative source for app configurations should remain in version control, typically Git. This enables code review, branching, and rollback. 

Recommended workflow: 

1. Developers commit app changes (including .conf changes for configuration‑only apps) into repository branches by app type and environment. 
2. A CI pipeline builds .spl/.tgz packages and uploads them to environment‑specific bucket paths, overwriting the existing archive for that logical app. 
3. Integration tests are run in dev/test clusters deployed via SOK. 
4. Upon approval, the pipeline promotes the same artifact (same filename) into higher‑environment paths (for example, from dev/ to test/ to prod/). 

This workflow aligns with the App Framework’s Etag‑based change detection and avoids manual uploads that can break traceability. 

Using polling versus manual triggers for controlled rollout 

By default, the App Framework polls remote storage at the specified interval to apply updates automatically. For sensitive environments such as production, this might be too implicit. 

For stricter control: 

• Set appsRepoPollIntervalSeconds: 0 for production CRs. 
• Use the namespace‑level or per‑CR ConfigMap (splunk-<namespace>-manual-app-update or splunk-<namespace>-r-name-configmap) to trigger updates only when planned. 

Example manual trigger: 

kubectl patch cm/splunk-prod-idxc-configmap \ 
  --type merge -p '{"data":{"manualUpdate":"true"}}' 

This pattern allows: 

• Continuous deployment in lower environments with polling enabled. 
• Change‑controlled deployment in production with manual triggers.

Coordinating premium app lifecycles 

Premium apps like Splunk Enterprise Security (ES) have additional constraints and supplemental components (for example, indexer TAs) that must be handled carefully. ES deployment via SOK uses the App Framework to install ES on SHCs and requires manual deployment of the indexer‑side TA to peers through the ClusterManager CR’s appSources. 

Best practices: 

• Treat ES and its supplemental TAs as a distinct core app domain with its own lifecycle and change process. 
• Validate compatibility with Splunk Enterprise and operator versions using official SOK and ES documentation prior to upgrades. 

Operator staging volume 

Configure a persistent volume for the operator pod to use as the app staging area instead of main memory. This reduces the risk of memory pressure when managing many or large apps. 

Example: 

kind: PersistentVolumeClaim 
apiVersion: v1 
metadata: 
  name: operator-volume-claim 
spec: 
  accessModes: 
    - ReadWriteOnce 
  resources: 
    requests: 
      storage: 8Gi 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: splunk-operator 
spec: 
  template: 
    spec: 
      containers: 
        - name: splunk-operator 
          volumeMounts: 
            - mountPath: /opt/splunk/appframework/ 
              name: app-staging 
      volumes: 
        - name: app-staging 
          persistentVolumeClaim: 
            claimName: operator-volume-claim 

Storage security and access 

SOK requires read‑only access to the app storage location and recommends securing connections with at least TLS 1.2. Where possible, use IAM roles, Managed Identities, or Workload Identity rather than static access keys in Kubernetes secrets (if running on AWS). 

Best practices: 

• Limit read access to specific buckets/containers and folders rather than entire storage accounts. 
• Avoid storage account‑level shared access keys if managed identities are available. 
• Rotate any static credentials regularly if they must be used. 

Monitoring, failure states, and rollback 

The App Framework reports health via per‑app status fields on the Splunk CR, covering download, copy, install, and (for clustered CRs) bundle‑push phases. It does not provide built‑in rollback or push‑style alerting, so operators must interpret these states, implement rollback through packaging and storage, and surface failures to an external monitoring stack. 

Other operational tips 

• Set realistic appsRepoPollIntervalSeconds values; very short intervals can lead to unnecessary load, while very long intervals slow updates. 
• Ensure consistent polling configuration across CRs of the same type in a namespace; mixing enabled and disabled polling for the same CR type can cause unexpected behavior. 

Ownership by app category 

Establish clear ownership across app categories, so responsibilities are unambiguous and aligned with how SOK centralizes governance while keeping namespaces and deployments isolated. 

• Platform / SRE Team (Splunk platform owners) 
  • Owns the SOK deployment itself (operator installation, CRD versions, namespaces, RBAC, secrets). 
  • Owns core platform apps and configuration‑only apps (indexes, authentication, RBAC, limits, macros shared platform‑wide). 
  • Owns the App Framework configuration in CRs (appRepo, volumes, appSources) and the layout, security, and lifecycle of remote app storage. 
  • Owns global rollout policy (when production bundles are pushed, which environments use polling vs manual triggers). 
  • Provides run‑books and standards for packaging, versioning, and naming of apps and TAs. 
• Data Engineering / Data Platform / Integration Teams 
  • Own technology add‑ons (TAs) and data onboarding pipelines (inputs, props/transforms, modular inputs). 
  • Validate TA versions and compatibility with source systems, mapping to Splunk best‑practice deployment targets (HF/indexer/SH). 
  • Propose TA changes via Git (PRs) into the platform‑managed repos; platform team reviews and merges changes that affect shared tiers. 
• Application / Domain / Line‑of‑Business Teams 
  • Own user‑facing apps (dashboards, saved searches, alerts, knowledge objects) deployed to SHCs or standalone SHs. 
  • Are responsible for business logic, KPIs, and UX of their apps, including data model usage and performance of their searches. 
  • Use the agreed CI/CD and app packaging patterns; do not change CR specs or SOK configuration directly. 
• Security / Compliance / Risk Teams 
  • Define guardrails for what can be deployed (approved source storage locations, allowed app types, security review of third‑party apps). 
  • Might require additional approval for high‑risk changes (for example, ES upgrades, apps with privileged scripts, or ingest‑actions affecting data routing). 

Approval flows and environments 

Define explicit approval paths so that “who can ship what where” is clear, and vary the strictness by environment. 

• CR spec changes (SOK topology, App Framework config): 
  • Proposed and implemented by Platform/SRE team only. 
  • Changes flow via GitOps (PR + review) and are typically subject to CAB/change‑control for production namespaces. 
• Core apps & configuration‑only apps: 
  • Owned by Platform; changes require at least Platform + Security/Compliance review for production, due to impact on all users and data. 
  • Promotion from dev → test → prod follows a central release calendar with regression testing. 
• TAs / data‑ingress configuration: 
  • Proposed by Data Engineering; reviewed/approved by Platform (and Security if external connectivity, credentials, or privileged code are involved). 
  • Promotion is gated on functional validation against representative data and performance impact on indexers/search heads. 
• User‑facing apps: 
  • Owned and approved within the domain team for lower environments (dev/test). 
  • For production, require at least: domain owner sign‑off + Platform review for operational impact (search load, storage, scheduling of saved searches/alerts). 
• Environment gates: 
  • Dev: low friction, fast iteration once automated checks pass; Platform sets global constraints (resource limits, polling frequency). 
  • Test/Staging: peer review and automated tests required; Platform might enforce additional checks for core/TA changes. 
  • Production: changes that affect shared tiers (indexers, SHC, MC, LM) require Platform and often Security/Compliance approval, with scheduled, coordinated deployments. 

Day‑to‑day responsibilities 

Clarify who builds, who deploys, who monitors, and who rolls back in daily operations. 

• Build and package 
  • App and TA authors (Platform, Data, or Domain teams) own code and .conf content and maintain app source in Git. 
  • They follow the packaging standard and ensure compatibility with supported Splunk and operator versions. 
• Deploy and operate 
  • Platform/SRE team owns CR definitions, App Framework configuration, and the SOK runtime (operator health, PVCs, storage integration, restarts). 
  • They control which buckets/paths are wired to which environments, and how polling/manual triggers are configured. 
• Monitor and alert 
  • An external monitoring team/stack (not the SOK‑managed Splunk deployment) monitors operator health, Splunk CR status for App Framework phases, and derived metrics such as failed or stuck apps and bundle‑push SLOs. 
• Roll back and remediate 
  • When a bad app deployment is detected, the owning team for that app (Platform for core/config, Data for TAs, Domain for user apps) leads remediation. 
  • Platform executes the technical rollback actions in storage/CRs, per the agreed rollback run‑book (republish previous archive or publish disabled version). 

Monitoring and alerting responsibilities 

To avoid Splunk‑monitoring‑itself anti‑patterns, SOK health is observed by an external solution. 

• What is monitored 
  • Operator pod health and logs. 
  • Splunk CR status for App Framework phases (download/copy/install/bundle‑push). 
  • Number of apps in failure or stuck states; SLOs for deployment latency (time from artifact upload to “all apps in install complete and bundle push complete”). 
• Who monitors 
  • An independent monitoring/observability platform such as Splunk Observability Cloud (which might ingest metrics/logs from Kubernetes and the operator but runs outside this SOK deployment). 
  • The team operating that platform owns alert definitions and first‑line triage and escalates to Platform/Data/Domain teams as appropriate. 

Rollback and change control 

Rollback is handled through packaging and storage, since the App Framework does not provide native removal or rollback. 

• Rollback mechanism (for all app types): 
  • Use stable app filenames and treat the remote object’s Etag as the version signal. 
  • Rolling back means overwriting the object at that path with a previously validated archive, then triggering App Framework reconciliation (poll or manual update). 
  • For urgent mitigations, publish a version of the app where app.conf disables the app while root cause is investigated. 
• Who can initiate rollback: 
  • Any team can request rollback for an app they own, but only Platform (or a designated CI/CD pipeline acting under Platform control) can modify production storage paths or CRs. 
  • Rollback decisions and communication are handled via normal incident/change‑management processes. 

Support boundaries 

Clarify where SOK and the App Framework end, and where customer responsibilities begin, linking back to documented limitations. 

• SOK / App Framework responsibilities 
  • Apply CR specs declaratively and manage underlying Kubernetes objects for Splunk Enterprise. 
  • Fetch, stage, and deploy app archives from the configured App Sources to the appropriate Splunk CRs. 
  • Maintain status codes for download/copy/install and (where applicable) coordinate bundle pushes. 
• Customer / internal teams’ responsibilities 
  • Design, package, and validate apps and TAs; ensure compatibility and security of their contents. 
  • Provide and secure external object storage (buckets/containers, IAM, TLS) for hosting app artifacts. 
  • Implement external monitoring, alerting, and incident response around operator and App Framework health. 
  • Perform manual app removals or complex remediation steps that are outside of App Framework capabilities and then align packages and storage so that state remains declarative and repeatable.