Skip to main content
Lantern Home

 

Splunk Lantern

Manage observability configurations as code with the Splunk Observability Cloud Terraform provider

  1. Last updated
  2. Save as PDF

 

As applications evolve beyond the early stages of development, their architectures become increasingly complex. This is especially true in continuous integration and continuous deployment (CI/CD) environments, where configurations change frequently. In fast-moving, large-scale environments, manually tracking every code change, infrastructure update, or observability configuration is not practical.

Terraform is a widely used Infrastructure as Code (IaC) tool developed by Hashicorp. It addresses these challenges by automating infrastructure management, integrating configurations into version control systems, and reducing manual intervention.

In this article, you'll learn about:

Terraform: overview and benefits

Instead of manually configuring infrastructure through user interfaces (UIs) or command-line interactions, Terraform allows for provisioning and management via code. This allows development teams to apply software development best practices to infrastructure management, improving maintainability and scalability. Terraform uses human-readable configuration files (*.tf) to define the desired state of infrastructure, while plugins, known as providers, allow integration with cloud platforms and services to define pieces of infrastructure. The Terraform Registry offers over 1,000 providers for AWS, Azure, Google Cloud Platform (GCP), Kubernetes, Splunk, and other services.

Terraform automates the infrastructure management process, reducing manual provisioning efforts that are error-prone and time-consuming. Teams can build, change, and manage infrastructure in a version-controlled, shared, and repeatable way all from a centralized Terraform file. This file can be committed and stored alongside existing code in tools such as GitHub or GitLab for safe and collaborative infrastructure management. Additionally, multiple cloud platforms can be managed from within the same Terraform file, eliminating the need to manually move between different configuration interfaces. Terraform's human-readable syntax simplifies configuration, and its state file maintains infrastructure changes for accurate tracking and management.

Terraform is designed for ease of use. After installing Terraform, users can initiate a configuration with a main.tf file and a few commands to initialize, preview, and apply their infrastructure plan.

Terraform and observability

Observability configurations, like infrastructure components, can be managed as code using Terraform. Instead of manually creating dashboards, charts, alerts, or cloud platform integrations through an observability UI, these resources can be defined, versioned, and deployed using Terraform. This approach offers several advantages, explained in the next few sections.

Standardization and consistency

Terraform configurations can be structured into reusable modules, allowing teams to define observability resources consistently across multiple services and environments. Standardized dashboards, alerts, and charts improve visibility and facilitate troubleshooting. Manual configuration introduces inconsistencies and increases the likelihood of errors, whereas Terraform ensures uniformity and prevents configuration drift.

Collaboration and improved maintenance

Managing observability as code provides version control, audit logs, and a structured change approval process. Unlike manual UI-based changes, Terraform configurations require check-in and approval through a version control system such as GitHub or GitLab. This ensures transparency, prevents conflicting changes, and enables controlled rollbacks if needed.

Scalability and efficiency

Terraform centralizes observability configurations alongside code repositories, reducing the need for manual updates through UI interactions. Instead of navigating through multiple dashboards to adjust alert thresholds, users can modify configurations in the Terraform codebase and redeploy. Terraform facilitates bulk updates across multiple environments, eliminating repetitive manual adjustments.

While managing dashboards and charts through Terraform might seem complex, it is highly beneficial in microservices architectures. Each microservice typically exists across multiple environments (for example, development, staging, production), and defining observability configurations in code enhances deployment speed and accelerates issue detection.

Security enhancements

Observability as code enhances security by centralizing credential management. Instead of distributing observability platform credentials across multiple team members, Terraform securely stores API keys and manages them within a controlled environment alongside other infrastructure secrets.

Creating observability resources using the Splunk Observability Cloud Terraform provider

Let’s see how to create observability resources using the Splunk Observability Cloud Terraform provider.

If you don’t already have Terraform installed, you must do that first. Check the Terraform documentation for tutorials.

  1. Create a new main.tf file in the application’s root directory or modify an existing Terraform file. Here is an example configuration for the Splunk Observability Cloud Terraform provider:

    clipboard_e40b90514a7cfdabb209061cedc12da52.png

  2. Define the required_providers block to declare provider dependencies, specifically the signalfx provider for Splunk Observability Cloud. You can think of the required_providers block like an import statement, while the following provider-specific block configures the signalfx provider.

    If your Splunk organization is located in us1 instead of the default us0, specify the api_url.

  3. Define resource blocks to create observability objects such as dashboards, detectors, or charts. Refer to the Splunk Observability Cloud Terraform provider docs under the Resources section for information on all available configurable resources.
  4. Run terraform init to initialize Terraform and install the Splunk Observability Cloud provider.

    clipboard_e678e96691f1d92059c654ab53d3cd3ae.png

  5. Run terraform plan to preview the actions Terraform will perform after the configuration is applied.

    clipboard_e9b437be80f15d5a6b5c9c330331c268c.png

  6. Run terraform apply to create your resources. In this example, we're creating a dashboard group and dashboard.

    clipboard_e331a49e7a2b191978937714df8ec239e.png clipboard_e6401bff06179a364ad09e2c8d91ee11d.png

  7. Verify that your resources have been successfully created in the Splunk Observability Cloud UI.

    clipboard_e04baca67dd56329b3db09b95a3f2aa37.png

  8. Although you've created the resources, these aren't too useful to you yet as there's no information within them that can help you on your observability journey. To add new resources or update existing ones, you can edit your main.tf file. The example below shows edits made to main.tf to add a chart resource and update the dashboard.

    clipboard_e182dd5ff767fbadb8d6281b3293adc8b.png

  9. Run terraform plan to preview the changes.

    clipboard_e8fc0ad06f4282fd042d28c660984d262.png clipboard_e999e9422f867e69ed0ea5517ae267116.png

  10. Apply the changes using terraform apply and verify the chart in the dashboard.

    clipboard_e92a4101562ab9f694656c6bce5016d13.png

Next steps

Whether you’re building out detectors and alerts, dashboards and charts, or integrating cloud services via Terraform, managing observability configurations as code can improve resource consistency, collaboration, and maintenance while increasing development and incident resolution speed.

If you’re new to Terraform, these resources might help you understand and implement this guidance: