Deploying identical add-ons with slightly different configurations

There are some situations where you may need to deploy the same add-on multiple times with slightly different configurations. One examples of this could be with the Splunk Add-on for Unix and Linux. Some inputs, like system logs, may be be enabled on all hosts. Other inputs like CPU and disk may only be needed on a subset of the hosts.  Others will be enabled on only some deployment clients based on the needs of the client machine's owners. It's clear from the documentation how to deploy this add-on with some inputs enabled within the configuration file local/inputs.conf.

Managing these situations complexities correctly simplifies the upgrade burden of Splunk and removes hte need for excess infrastructure 

Configuration management topics in Splunk Docs

The Splunk Enterprise Admin Manual provides excellent coverage About configuration files. Here's some critical topics within that section and how they relate to this discussion.

• Configuration file directories - Editing to ensure add-on upgradability requires a clear understanding of the risks.

• Configuration file structure - Cloning configurations requires a confident grasp on stanzas.

• Configuration file precedence - Cloning and editing configuration files requires a clear understanding of how Splunk Enterprise reconciles the duplicate configurations.

Here are some additional topics in other manuals that are important to know about.

• Use btool to troubleshoot configurations - btool shows what the blend of configuration looks like if Splunk reloads the configuration.

• Updating Splunk Enterprise Instances - this manual describes how the deployment server and deployment clients work to manage updates to Splunk Enterprise instances.

• Where to place the scripts for scripted inputs - this topic describes how to manage scripted inputs, and how to ensure that relative paths are calculated correctly when Splunk Enterprise computes the fully qualified path to the script.

• About installing Splunk add-ons - This scenario focuses on special considerations for distributed deployments and Splunk Cloud topologies.

• Click here for a video about the suggested naming convention for Splunk apps and add-ons.

Potential solutions for deploying multiple copies of an add-on with different configurations

Here are some options for deploying multiple copies of an add-on with different configurations, where the options work well, the complexity or level of effort to implement, and where the options run into challenges.

Brute force

• Approach: Deploy the add-on with a configuration that is common to all clients, then manually add configurations to the individual clients that need deviations.

• Level of effort: Trivial

• Gotchas: Hard to find and manage deviations at scale. Requires adjustments to accommodate the relative paths for scripted inputs.

Clones

• Approach: Completely duplicate the app into a another app. Customize each clone to the specific needs of its target client. Deploy each specific clone to its target client. Each clone now contains all of the substance of the original app, but with your local additions. Only deploy the clones, not the original app, since it will be redundant and unused.

• Level of effort: Implementation is moderate. Since upgrading each clone requires duplicating all over again, such recurring maintenance may be undesirable amount of work. Scripted inputs, relatively defined, will work natively.

• Gotchas: Upgrades can be error prone and labor intensive, since you effectively recreate each clone. You may also need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.

Parent and children

• Approach: Deploy the app with a configuration that is common to all clients and call it the 'parent'. Create 'children' that contain only the deviations needed for groups of clients. Deploy the children to their targets.

• Level of effort: Moderate to large. Since all configuration is centralized, this approach scales well to clients, but becomes difficult to corral as the number of children increases (get it?).

• Gotchas: Sharing scripted inputs in this model causes many other changes in order to work and therefore is not recommended. Also requires a naming convention to identify all the children and associate them with their parents. For example, when creating a variation of the Splunk_TA_windows from Splunkbase for my AWS environment, I'll name the variation Splunk_TA_windows-aws, which identifies it as a child of the parent app Splunk_TA_windows.

Symbolic links

• Approach: Create symbolic links (symlinks) of all folders except local into a new app. In the new app, populate the local folder with the desired configuration for the target clients. The new app will now contain all of the substance of the original app, but with your local additions. Deploy only the new app, not the original app since it will be redundant and unused.

• Level of effort: Implementation is complex. Upgrading is trivial.

• Gotchas: There are passionate debates about symlinks because of the obfuscation introduced by the symlink abstraction. You may need a naming convention to identify the source and its various clones. Hard-coded references to the original app name will cause errors, since the original app is not deployed.