Skip to main content
Lantern Home

 

Splunk Lantern

Evolving your agent ecosystem between Splunk AppDynamics and Splunk Observability Cloud

  1. Last updated
  2. Save as PDF

This article explains how to transition your Splunk AppDynamics and Splunk Observability Cloud monitoring agent ecosystem using the AppDynamics Combined Agent. 

The Combined Agent bundles both AppDynamics agent code and Splunk OpenTelemetry agent code into a single package. This innovative approach addresses one of the most significant challenges organizations face when considering observability platform changes: the complexity and risk of re-instrumenting applications.

The AppDynamics Combined Agent bundles AppDynamics instrumentation with Splunk OpenTelemetry (OTel) capabilities, allowing you to operate in three distinct modes: AppD mode, dual mode, or Splunk mode. This provides a low-risk, predictable path for organizations to:

  • Continue using AppDynamics with no changes (AppD mode)
  • Evaluate Splunk Observability Cloud alongside AppDynamics (dual mode)
  • Migrate fully to Splunk Observability Cloud (Splunk mode)

This approach preserves your existing instrumentation investment while providing flexibility to adopt new observability capabilities as your needs change, while eliminating the traditional "rip and replace" challenges associated with changing observability platforms.

Key benefits include:

  • Reduced operational overhead: No need to maintain separate agent deployments or coordinate complex migration projects
  • Risk mitigation: Test new observability capabilities alongside existing monitoring without disrupting production systems
  • Flexibility: Choose the right observability backend for each use case without being locked into a single solution
  • Simplified evaluation: Compare both platforms using the same instrumentation points

The Splunk AppDynamics Hybrid Agent (an AppD agent that speaks OpenTelemetry Protocol) is supported and maintained for the AppDynamics native experience. However, it is not recommended and will not be used as a source of data for Splunk Observability Cloud. The Combined Agent approach described in this article is the recommended path forward.

Prerequisites

  • Existing Splunk AppDynamics instrumented environment
  • Access to Splunk Observability Cloud
  • Latest AppDynamics Combined Agent (check the release notes for the latest supported versions for Java, Node.js, and .NET)

Understanding the agent ecosystem

Splunk Observability Cloud agents

  • Based on OpenTelemetry (OTel) standards
  • Open-sourced with additional Splunk enhancements through the Splunk Distribution of the OpenTelemetry Collector
  • Communicates using the OpenTelemetry Protocol (OTLP)
  • Supports a vast range of environments and is unmatched for cloud-native, microservices architectures

Splunk AppDynamics agents

  • Proprietary agents purpose-built for AppDynamics
  • Communicate over a closed protocol
  • Best suited for traditional three-tier/n-tier applications and monolithic systems
  • Support both SaaS and on-premises deployments

The AppDynamics Combined Agent

The Combined Agent bundles both AppDynamics Agent code and Splunk OpenTelemetry Agent code into a single package. This enables:

  • Deployment using existing AppDynamics agent processes
  • Support of multiple modes of operation
  • Easy mode switching between backends
  • Simplified evaluation and migration paths
  • Validated compatibility between both agent types

Languages supported initially by the AppDynamics Combined Agent are:

  • Java
  • Node.js
  • .NET

Both Splunk Observability Cloud and Splunk AppDynamics agents will continue to be developed, maintained, and supported to provide continuity for existing users. In future, it's expected that the agents will undergo low impact changes and evolution toward greater use of OpenTelemetry.

How to use Splunk software for this use case

    The Combined Agent supports three modes:

    Mode Description Use Case Behavior details
    AppD mode (Default) Only AppDynamics agent is active; data sent to AppDynamics backend Standard AppDynamics monitoring
    • The Splunk OpenTelemetry component remains dormant
    • Zero impact on existing AppDynamics monitoring
    • Standard AppDynamics agent behavior maintained
    • Possibly a small increase in the agent package size on disk; runtime memory usage is not affected
    Dual mode Both agents active; data sent to both backends Evaluation and testing (not recommended for production due to increased overhead)
    • Both instrumentation engines activate simultaneously
    • Telemetry data flows to both AppDynamics and Splunk Observability Cloud
    • Enables side-by-side comparison of both platforms
    • Higher overhead due to dual instrumentation, including increased resource consumption and impact to operations
    Splunk mode Only Splunk OTel agent is active; data sent to Splunk Observability Cloud Full migration to Splunk Observability Cloud
    • AppDynamics instrumentation deactivates
    • Only OpenTelemetry-based instrumentation runs
    • Data flows exclusively to Splunk Observability Cloud
    • The agent deregisters itself from AppDynamics
    • Provides pathway to eventually replace with standard Splunk OTel agents

    Step-by-step migration guide

    For a full demonstration of this process, see the recording of the .conf presentation that this article is based on (the demo starts around 19:30).

    Step 1: Upgrade AppDynamics agents

    Update all AppDynamics language agents to the latest versions that include Combined Agent capabilities (check the release notes for the latest supported versions for Java, Node.js, and .NET). Key points to be aware of include:

    • The agent package name remains the same - only the version changes
    • The upgrade follows standard AppDynamics agent update procedures
    • No immediate behavioral changes occur after upgrade; the agent continues operating in AppD Mode by default
    • Leave cluster agent and machine agent intact during this process
    • For Java, copy the new agent files to the appropriate location and ensure the Java instrumentation is added to the command

    Step 2: Install Splunk OpenTelemetry Collectors

    Deploy Splunk OpenTelemetry Collectors throughout your infrastructure. The collector is essential for any OpenTelemetry-based instrumentation ecosystem and serves as the telemetry pipeline for data flowing to Splunk Observability Cloud.

    Deployment considerations include:

    • The collector is published as a Docker image by Splunk and OpenTelemetry
    • Control it with a Docker Compose configuration
    • Add environment variables to control the configuration
    • Deploy collectors in Kubernetes environments using the recommended patterns
    • For bare metal or virtualized environments, install collectors on each host
    • The collector handles infrastructure monitoring data, in addition to application traces

    Step 3: Configure agent mode

    Switch the Combined Agent to your desired mode of operation. The mode switch is controlled by a single configuration flag and can be set through environment variables or configuration files.

    Option A: Dual mode (for evaluation)

    Enable dual mode to send data to both Splunk AppDynamics and Splunk Observability Cloud simultaneously. This requires adding one mode switch flag to your configuration.

    Important considerations for dual mode include:

    • Increased resource overhead (CPU and memory)
    • Not recommended for production environments without thorough testing
    • Useful for validating data parity between backends
    • Some high-performance, low-latency applications might not tolerate the additional overhead
    • Traditional applications with standard request flows might not show significant performance differences
    • Cold start time impact is most noticeable in dual mode

    When dual mode is active, you should see:

    • Log messages indicating the dual mode has been picked up
    • Information about the Java OTel agent starting
    • Confirmation that both the OTel agent and AppDynamics agent have started
    • The collector will show traces being generated

    Option B: Splunk mode (for full migration)

    Enable Splunk mode to send data exclusively to Splunk Observability Cloud. Again, this is just one flag change.

    When Splunk mode is active, you should see:

    • Fewer log messages compared to dual mode
    • Only the OTel agent startup confirmation
    • The AppDynamics node will no longer appear in the AppDynamics Controller (the agent deregisters itself)

    Option C: AppD mode

    If you decide that you don't want to migrate to Splunk Observability Cloud, use the new agents and don't enable dual mode or Splunk mode. The Combined Agent in AppD Mode behaves exactly like the traditional AppDynamics agent. In the future, when AppDynamics' OTel capabilities are improved, you could switch to Splunk mode while continuing to use AppDynamics, or even use OTel agents directly with AppDynamics.

    Step 4: Validate data flow

    For Splunk AppDynamics:

    • Verify that the agent version appears correctly at the node level in the AppDynamics Controller
    • Check that business transactions and flow maps populate as expected
    • Monitor node-level metrics (software level and hardware level)

    For Splunk Observability Cloud:

    • Verify that traces appear in Splunk Application Performance Monitoring
    • Check infrastructure metrics from the collector
    • Validate service maps and dependencies
    • View database calls, traces, and spans
    • Confirm that the collector is receiving and forwarding trace data
    • Switch between collector view and associated software view

    Step 5: Complete the migration

    After you've validated that Splunk Observability Cloud is receiving all necessary data:

    1. Switch all agents to Splunk mode (OTel mode)
    2. Remove or disable AppDynamics-specific components that are no longer needed
    3. Optionally, replace Combined Agents with standard Splunk OTel agents for a simplified deployment

    Future roadmap

    Expanding language support

    Additional languages and agents planned for Combined Agent support:

    • Python
    • Machine Agent

    OpenTelemetry contributions

    Splunk continues to contribute AppDynamics-inspired features to OpenTelemetry upstream. These contributions ensure that capabilities unique to AppDynamics become available to the broader OpenTelemetry ecosystem and can be used by other vendors. Features not yet ready to be accepted by the OpenTelemetry community are added to Splunk distribution open-source repositories in preparation for that process.

    AppDynamics for OpenTelemetry

    The AppDynamics for OTel feature set will evolve to support true OTel ingestion into AppDynamics, particularly for languages not natively instrumented by AppDynamics agents. This creates a feedback loop where OTel agents can power AppDynamics-specific features.

    Additional resources

    The content in this article comes from a .conf presentation, one of the thousands of Splunk resources available to help users succeed.

    In addition, these resources might help you understand and implement this guidance: