Accelerating an implementation of OpenTelemetry in Splunk Observability Cloud
This Splunk Observability Cloud OpenTelemetry accelerator is a comprehensive guide designed for Splunk Observability Cloud customers who are new to OpenTelemetry and want to set up telemetry in their on-premises or cloud infrastructure using the Splunk Distribution of the OpenTelemetry Collector and instrumentation libraries.
The accelerator provides a high-level overview of key topics that can help you to set up telemetry. Depending on your infrastructure, you can pick and choose what makes the most sense for your organization - for example, if you don't have Amazon ECS Fargate, you can skip it.
This Splunk Observability Cloud OpenTelemetry accelerator is available as an engagement with Splunk Professional Services. If you do not feel comfortable completing these processes on your own, or would like hands-on training with any of the concepts and processes included in this article, contact our Professional Services experts.
Before beginning, ensure you have administrative access to the infrastructure for the OpenTelemetry collector installation, as well as access to the application code for instrumentation and custom metrics generation using instrumentation libraries. Then, you and your team will work through the following tasks:
- Splunk Distribution of the OpenTelemetry Collector installation with advanced parameters
- Splunk Distribution of the OpenTelemetry Collector helm chart installation (for Kubernetes)
- Config management tools for the Splunk Distribution of the OpenTelemetry Collector installation
- Splunk Distribution of the OpenTelemetry Collector installation for Amazon ECS Fargate
- Splunk Distribution of the OpenTelemetry Collector advanced configuration
- Filtering, routing and transforming telemetry data using the OpenTelemetry Collector
- Splunk Distribution of the OpenTelemetry Collector gateway setup
- Log collection using the Splunk Distribution of the OpenTelemetry Collector (only applicable for Kubernetes use cases)
- Application instrumentation
- Generate custom metrics using instrumentation libraries for one of the supported languages
Tasks
Splunk Distribution of the OpenTelemetry Collector installation with advanced parameters
- Learn about deployment modes for the Splunk Distribution of the OpenTelemetry Collector.
- Learn about supported processor architecture for the Splunk Distribution of the OpenTelemetry Collector.
- If you're using Linux, install the Splunk Distribution of the OpenTelemetry Collector using the installer script and take note of the following guidance:
- Learn about about available parameters of the installer script for Linux.
- Understand the location of the Splunk Distribution of the OpenTelemetry Collector binaries, so you can set up your internal artifactory if required:
- If you have a requirement to set up the Splunk Distribution of the OpenTelemetry Collector manually (without using the installer script), read a step-by-step guide on how to do this.
- Learn about about available parameters of the installer script for Linux.
- If you're using Windows, install the Splunk Distribution of the OpenTelemetry Collector using the installer script and take note of the following guidance:
- Learn about available parameters of the installer script for Windows.
- Learn about the location of Splunk OpenTelemetry Collector msi files, so you can set up your internal artifactory if required.
- If you have a requirement to set up the Splunk Distribution of the OpenTelemetry Collector manually (without using the installer script), read a step-bystep guide on how to do this.
- If required, learn how to set up a proxy for the Splunk Distribution of the OpenTelemetry Collector.
Splunk OpenTelemetry Collector helm chart installation (for Kubernetes)
- Review the different components of the Splunk Distribution of the OpenTelemetry Collector for Kubernetes helm chart and its functionality, for example:
- Daemonset
- ClusterReceiver
- Gateway (optional)
- Configmaps
- Secrets
- Operator (required for zero-code instrumentation)
- Review the advanced configurations to understand the different configuration options available in the values.yaml file.
- Validate that telemetry data is available in Splunk Observability Cloud.
- If required, set up additional receiver configuration using receiver_creator so you can collect metrics from dynamically instantiated pods.
- Customize your pipeline to fit your use case, for example:
- Adding an additional processor in the pipeline to filter data
- Adding a new receiver to the pipeline
Config management tools for the Splunk Distribution of the OpenTelemetry Collector installation
Splunk Distribution of the OpenTelemetry Collector installation for Amazon ECS Fargate
- Deploy the collector on ECS.
- Learn about metrics received by the default configuration.
- Learn about the default configuration used in ECS collector deployment.
- Learn how to use custom configuration for the Splunk Distribution of the OpenTelemetry Collector on ECS.
- Learn how to instrument the application in ECS using this example for Java services.
Splunk Distribution of the OpenTelemetry Collector advanced configuration
- Learn about the different OpenTelemetry Collector distributions:
- Learn about the different components of the OpenTelemetry Collector configuration:
- Learn about available environment variables for Splunk Distribution of the OpenTelemetry Collector.
- Learn about monitoring Splunk Distribution of the OpenTelemetry Collector using internal metrics.
-
- Learn about exposed ports and endpoints by the Splunk Distribution of the OpenTelemetry Collector.
- Learn about available options for secrets management in the Splunk Distribution of the OpenTelemetry Collector:
- Set up secret management using your preferred method.
Filtering, routing and transforming telemetry data using the OpenTelemetry Collector
- Learn about filtering data using the filter processor.
- Learn about routing data using the routing processor.
- Learn about adding additional metadata to the existing telemetry data using the attributes processor.
Splunk Distribution of the OpenTelemetry Collector gateway setup
- Set up the Splunk Distribution of the OpenTelemetry Collector gateway mode.
- Learn about sizing and scaling of the gateway.
Log collection using the Splunk Distribution of the OpenTelemetry Collector (only applicable for Kubernetes use cases)
- Learn about log collection using Splunk Distribution of the OpenTelemetry Collector filelog receiver.
- Learn about the various available operators while using the filelog receiver.
- Learn about including or excluding certain log files.
- Learn about common use cases, such as:
Application instrumentation
Generate custom metrics using instrumentation libraries
- Generate custom metrics from applications/services:
Next steps
Successfully implementing OpenTelemetry in your infrastructure is a significant achievement. To ensure a seamless transition to ongoing operations and to maximize your investment, consider the following next steps:
- Conduct a project wrap-up meeting
- Organize a final meeting with all stakeholders to review project deliverables.
- Summarize high-level achievements and ensure alignment on next steps.
- Highlight key metrics, insights, and outcomes.
- Review and document key learnings
- Consolidate and store documentation created during the implementation in a central place that's accessible to all stakeholders.
- Perform knowledge transfer
- Schedule dedicated sessions to train your team on:
- Using and maintaining the dashboards and telemetry data collected.
- Updating OpenTelemetry Collector configurations as requirements evolve.
- Troubleshooting common issues with OpenTelemetry agents and integrations.
- Schedule dedicated sessions to train your team on:
- Establish an ongoing monitoring strategy
- Define a regular cadence for reviewing and optimizing telemetry data.
- Assign responsibility for monitoring and maintaining your observability infrastructure.
- Document a scaling plan for observability as your infrastructure grows.
- Engage with Professional Services
- Consider periodic reviews with Splunk Professional Services to ensure your implementation aligns with best practices incorporates new features of Splunk Observability Cloud.
- Reach out to Professional Services for advanced use case consultations, hands-on training, or further customization.
By following these steps, you can ensure the successful integration of OpenTelemetry into your observability strategy and maximize the value of your investment in Splunk Observability Cloud.
Additional resources
Splunk Professional Services can provide hands-on Kubernetes guidance for you and your team. Click here to learn more about working with Professional Services.