Using service sandboxes in Splunk ITSI

Get started with service sandboxes

Open the ITSI interface, navigate to Configuration on the top horizontal toolbar, then Service Monitoring, and click Service Sandboxes in the drop down menu. From there you can navigate an organized list that shows all the sandboxes you or your team have previously created.

Create or import content

When creating or importing content, you have a few options:

• Create a New Service: Choose this if you’re testing out some services but aren't ready to link a service template or dependencies.
• Create from a Service Template: Choose this option if you have existing service templates to use to map out your service tree. This option is preferred if you want to simulate your sandbox health score with entity filtering rules and KPIs.
• Import from CSV: Choose this option if you worked with a Splunk professional services resource, services partner, or have an existing service decomposition CSV export.

To start configuring a new sandbox environment, click Create on the bottom right.

Do the following when configuring the sandbox:

• Name the sandbox: Choose a unique but categorically meaningful name so you and others can easily identify it.
• Add a description: Fill in the purpose of your sandbox.
• Services: Select the services you want to include in the environment.
• Permissions: Define the user roles and permissions for accessing and modifying the sandbox.

After you have completed the configuration, review your settings and click Create.

Set up your service tree

To set up your service tree:

1. Create the tree:
   1. Create a hierarchical structure of your services, defining parent and child relationships.
   2. Add service templates with KPIs and entity rules to the respective services.
2. Simulate service health scores:
   1. If creating a sandbox with a service template, you can understand the impact on the service health score by testing out the simulation of KPI severity and importance. Before publishing your service tree to the Service Analyzer, simulate the service health scores. This simulation helps you understand how the child services impact the parent service.
   2. By performing this simulation, you can catch potential issues early and make necessary adjustments.
3. Configure settings:
   1. There are additional general settings that you can configure on each service. First, choose whether the service should be enabled/disabled after going into Production.
   2. Next, configure the team level visibility for that service. 
4. Review and save:
   1. Review the results and save your work as a draft so you can share it with teammates for collaborative review. This allows you to further refine your configurations if needed.
   2. After you finalize it, publish the service tree to the Service Analyzer.

This process ensures that potential problems are addressed before they impact the live environment, thereby enhancing the reliability and performance of your IT services.

Pre-publish validation

To save progress in a sandbox, click Save as Draft. At this point, the sandbox conducts a comprehensive list of validation steps that scrutinizes your data and configurations to ensure that no errors exist that could disrupt services in production.

• Scan for errors: When you save, the system scans for any missing data, coding errors, and configuration issues that may affect performance or the sandbox outcome.
• Check compatibility: The system verifies that all components in the sandbox are compatible, fully functional, and available with the rest of your production environment.
• Performance: The system validates that all configurations meet the required efficiency standards without straining resources.

After completing the save draft step, but before publishing, the system undergoes another series of checks. The double layer of validation serves as a failsafe and gives you a chance to review potential issues before the final version goes live:

1. Revisit errors: The review revisits all previous checks to catch any errors possibly overlooked.
2. Check permissions: Permissions are viewed to ensure only authorized users can make final changes for publishing.
3. Final resource capacity: The capacity of resources are checked to ensure publishing doesn't cause unnecessary strain.

By incorporating these validation steps, the service sandbox helps reduce the risk of publishing issues to production and thereby enhances service setup.

Permissions and availability

Two main ITSI roles have different types of access:

• itoa_admin: Create services, publish services to Service Analyzer, reset and revert services from Service Analyzer back into service sandbox.
• Itoa_team_admin: Create services, save as draft, read only permission once a sandbox has published to Service Analyzer.

Troubleshooting common issues

The service sandbox feature in Splunk ITSI is a powerful capability for accelerating service configuration and optimization while minimizing risks to the production environment. If you encounter any challenges, refer to these troubleshooting options:

• Access denied: Ensure you have the appropriate permissions to access and manage the sandbox.
• Service sync issues: Verify that the services included in the sandbox are properly configured and synchronized with the ITSI environment.
• Performance degradation: Monitor the performance of services within the sandbox to identify and address any potential issues.

Next steps

Now that you’ve mastered these service decomposition best practices, what previously took a significant amount of time (even with professional services) will now decrease.

We look forward to hearing your feedback as we continue to improve service onboarding, decomposition, and getting started with Splunk products. Do not hesitate to reach out to your Splunk account team for additional feedback and support. For more detailed step-by-step instructions on how to set up your service sandboxes and additional granular capabilities, please refer to Splunk Docs.