A best practice for establishing a stable and reliable production Splunk environment is to set up a workflow that includes individual sandboxes for development and innovation, a lab environment for testing, and a safe push to production once things are ready.
Encouraging a healthy sandbox culture for your Splunk team ensures that your innovators have the latitude to try new things without disrupting what already works, or each other.
Note: This answer applies to Splunk Enterprise and Splunk Cloud.
How setting up a sandbox is a best practice for a healthy workflow
A local sandbox is a safe place for you to innovate and develop new ideas. The best sandbox is a stand-alone instance used by one person. Everyone on your Splunk team should have their own sandbox so they feel safe to take risks and learn. With your own sandbox, you'll not be afraid to start over if you need to.
A lab environment is where you can test features before bringing them to production. A lab environment should mirror your production environment and have access controls that support your testers and safeguard your production environment.
How to get started with sandboxes on Splunk Enterprise
Set up a sandbox! We recommend using Docker as your platform for setting up a sandbox because of how easily and rapidly it enables you to make mistakes, clean up, and start over. See our blog Hands on Lab: Sandboxing with Splunk (with Docker) for instructions.
Set up a lab! Set up a non-production lab to validate more complex and distributed features before bringing them to production.
How to get started with sandboxes on Splunk Cloud
Because Splunk Cloud is a SaaS service, you may not have access to anything but your production environment. Here is are a few ways to setup a sandbox or lab environment.
Create a sandbox app. This app can be hidden from view for all users but the developers (data governance!). As development work progresses, searches, reports, field extractions, etc, can each be moved into their production counterpart.
Create a sandbox app, then create a new sandbox index within that app. Data that is not yet ready to be moved into production can be sent here (and easily deleted). Once ready, you can change the target index at the universal forwarder, no longer pointing to the sandbox index, but now pointing to the production index.
Set up a stand-alone Splunk Enterprise instance. Splunk provides a free download of Splunk Enterprise. You can deploy this as a stand-alone deployment as a sandbox, and later move configurations you want to keep into your Splunk Cloud environment.