Skip to main content
Lantern Home
 
 
Splunk Lantern

Accelerating an implementation of Kubernetes in Splunk Observability Cloud

  1. Last updated
  2. Save as PDF

 

This Splunk Observability Cloud Kubernetes accelerator is a comprehensive guide designed to help organizations at the beginning of their Kubernetes observability journey. By leveraging best practices in telemetry setup and OpenTelemetry integration, this accelerator provides a structured framework for monitoring your Kubernetes environment using Splunk Observability Cloud and the Splunk platform.

The accelerator emphasizes actionable steps to set up Kubernetes monitoring, ensuring data from infrastructure, logs, and applications is seamlessly ingested and analyzed within the Splunk Observability Cloud ecosystem.

This Splunk Observability Cloud Kubernetes accelerator is available as an engagement with Splunk Professional Services. If you do not feel comfortable completing these processes on your own, or would like hands-on training with any of the concepts and processes included in this article, contact our Professional Services experts.

Before beginning, ensure you have admin access to your Kubernetes cluster. Then, you and your team will work through the following tasks:

  • Planning: Determine stakeholders, scope, and administrative requirements.
  • Kubernetes infrastructure monitoring: Configure a Kubernetes cluster, including the installation of the OpenTelemetry helm chart and validation of telemetry data in Splunk Observability Cloud.
  • OpenTelemetry agent metrics onboarding: Set up a custom receiver using the receiver creator and configure the helm chart pipeline for advanced use cases such as filtering data, or adding processors and new receivers.
  • OpenTelemetry collector installation and configuration: Install and configure a log operator to parse logs with features like multi-line configuration and JSON parsing.
  • APM auto instrumentation: Implement auto-instrumentation for an application in a supported language, and demonstrate zero-configuration instrumentation for another application.
  • Custom content creation: Build custom dashboards tailored to specific use cases and metrics. Design and implement custom detectors based on Kubernetes metrics to monitor critical thresholds and events.
  • Splunk Log Observer Connect configuration: Set up a Splunk Log Observer Connect instance to enable viewing of logs from Splunk Cloud Platform or Splunk Enterprise in Splunk Observability Cloud.
  • Out-of-the-box (OOTB) content review: Walk through prebuilt Kubernetes dashboards, auto detectors, and ingested telemetry data.

Tasks

Planning

When planning the implementation of Kubernetes observability, you should consider the following steps to set yourself up for success:

  • Determine the individuals or teams responsible for leading and contributing to the observability project.
  • Typical stakeholders include:
    • A central observability team or designated observability administrators (if available).
    • An architect who understands the high-level architecture of your Kubernetes environment and can provide setup guidance.
  • Clarify what your organization needs from your observability setup, including:
    • The type of telemetry data you want to collect.
    • Requirements for dashboards.
    • Requirements for detectors.
    • The Kubernetes cluster setup that will be used for the project.
    • Sufficient permission on the Kubernetes cluster fort agent installation for all involved personas.
    • The applications or services that will be used for auto-instrumentation or zero-configuration instrumentation.
    • Tag naming convention for various telemetry data.
  • Define clear objectives and set realistic delivery timelines for the project.
  • Schedule dedicated working sessions for each stage of the implementation, ensuring all relevant stakeholders can participate.
  • Determine documentation requirements and decide where project documentation will be stored.
  • Set up a reporting cadence to track progress and resolve blockers promptly.

Kubernetes infrastructure monitoring

  • Review the different components of the Splunk OpenTelemetry Collector for Kubernetes helm chart and its functionality, for example:
    • Daemonset
    • ClusterReceiver
    • Gateway (optional)
    • Configmaps
    • Secrets
    • Operator (required for zero-code instrumentation)
  • Review the advanced configurations to understand the different configuration options available in the values.yaml file.
  • Check resource availability.
  • Discuss the sizing of the Collector.
  • Validate that telemetry data is available in Splunk Observability Cloud.

OpenTelemetry agent metrics onboarding

  • Determine which supported monitors will need to be configured for metrics ingestion.
  • Set up additional receiver configuration using receiver_creator to collect metrics from dynamically instantiated pods.
  • Customize your pipeline to fit your use case, for example:
    • Adding an additional processor in the pipeline to filter data
    • Adding a new receiver to the pipeline

OpenTelemetry collector installation and configuration

APM auto-instrumentation

Custom content creation

Splunk Log Observer Connect configuration

  • Configure Splunk Log Observer Connect to view log data from Splunk Cloud Platform or Splunk Enterprise in Splunk Observability Cloud. Learn more at Getting started with Log Observer Connect.
  • Learn how related content works to allow seamless experiences across various telemetry data.

OOTB content review

  • Explore prebuilt Kubernetes dashboards, navigators, and detectors. Learn more at Monitor Kubernetes.

Next steps

Completing your Kubernetes observability implementation is a significant milestone. To ensure a smooth transition to ongoing operations and optimize your investment, follow these next steps:

  • Conduct a project wrap-up meeting
    • Organize a final meeting with all stakeholders to review project deliverables.
    • Summarize high-level achievements and ensure alignment on next steps.
    • Highlight key metrics, insights, and outcomes.
  • Review and document key learnings
    • Consolidate and store documentation created during the implementation in a central place that's accessible to all stakeholders.
  • Perform knowledge transfer
    • Schedule dedicated sessions to train your team on:
      • Using and maintaining the dashboards and detectors created.
      • Updating Splunk Log Observer Connect configurations as requirements evolve.
      • Troubleshooting issues with OpenTelemetry agents and Kubernetes integrations.
    • Provide walkthroughs of any custom pipelines or advanced configurations implemented during the project.
  • Establish an ongoing monitoring strategy
    • Define a regular cadence for reviewing and optimizing dashboards, detectors, and telemetry data.
    • Assign responsibility for monitoring and maintaining your observability infrastructure.
    • Document a plan for scaling observability as your Kubernetes environment grows.
  • Engage with Professional Services
    • Consider periodic reviews with Splunk Professional Services to ensure your implementation aligns with best practices and incorporates new Splunk Observability Cloud features.
    • Contact Professional Services for advanced use case consultations, hands-on training, or further customization.

Additional resources

Splunk Professional Services can provide hands-on Kubernetes guidance for you and your team. Click here to learn more about working with Professional Services.