Part of a successful Splunk implementation is establishing regular backups. You'll need to identify backup and restore points, and make regular backups of your Splunk configuration files to ensure system continuity in case of a failure, outage, or mistake.
Splunk backup policy best practices
$SPLUNK_HOME/etc/ directory and its subdirectories contain all the settings for your Splunk installation—including saved searches, user accounts, tags, and custom source type names—and all apps.
Back up to a separate location
Backing up to a separate location is a best practice for resiliency. Backing up to a different disk or mount than Splunk is installed on can reduce single points of failure.
Back up single points of failure
Back up any single points of failure, such as a single indexer, single instance deployment, or single search head, and utility tier resources, such as the deployment server, deployer, master node, or license server.
Back up at least one search head cluster (SHC) member periodically
As a best practice, periodically back up the SHC state to ensure you can restore knowledge objects to their current state in case of a catastrophic failure. For details about what to back up on the SHC and how, see Back up and restore search head cluster settings.
Implement some form of version control
Having some way of preserving state between versions is a best practice so you can roll back in case of an error.
- Standard. Scripted input that kicks off a specific diagnostic (or just an
$SPLUNK_HOME/etc/directory) and cleans old copies to prevent filling up the file system. See Generate a diagnostic file.
- Intermediate. Scripted input that checks into a source control system, such as Git.
- Advanced. A custom solution using source control, or CI/CD pipelines, for managed restoration.