Skip to main content
Lantern Home

 

Splunk Lantern

Setting up Azure Event Hubs in Data Manager

  1. Last updated
  2. Save as PDF

 

The Azure Event Hubs connector for Splunk Data Manager simplifies data onboarding by enabling seamless ingestion of event streams into Splunk Cloud Platform.

Azure Event Hubs is designed to handle large-scale data streams, processing millions of events per second. By integrating with Splunk Data Manager, organizations can efficiently capture continuous data flows, like logs or sensor information, and enhance their analytics capabilities.

The Data Manager Azure Event Hubs connector introduces several powerful features to streamline data ingestion. One major improvement is automatic partition detection, which eliminates the need for manual configuration. When you create an input, Data Manager automatically identifies the available partitions and assigns collectors accordingly. This ensures efficient data collection and reduces setup time.

Key features of the connector include:

  • Partition load balancing: The connector uses the Azure Event Hubs Go SDK to automatically distribute data partitions across multiple collectors. This load balancing mechanism improves system resiliency; if a collector fails, others automatically take over its partitions. This ensures uninterrupted data flow and a more stable system overall.
  • Checkpointing: Checkpointing tracks the exact position of each consumer within a partition. If a disconnection occurs, the connector resumes data collection from the last checkpoint, preventing gaps or duplicate data ingestion.
  • Authentication: The connector supports a secure authentication method that enhances security while simplifying credential rotation. This helps organizations maintain strong access controls without adding complexity to their workflow.
  • Event building and error handling: The connector maps event attributes - such as the body and properties - directly to log records. This ensures compatibility and maintains correct formatting, reducing the risk of data inconsistencies.
  • Exporter: The connector uses the Splunk HEC exporter with a /raw endpoint to properly format and ingest complex event data into Splunk Cloud Platform. This ensures that all collected data is structured for efficient analysis.

This article provides guidance for configuring the Azure Event Hubs Data Input in Data Manager, as well as migrating from the Splunk Add-on for Microsoft Cloud Services to Data Manager.

Configuring the Azure Event Hubs Data Input in Data Manager

  1. Open Data Manager.

    unnamed - 2025-02-19T115929.777.png

  2. Select Microsoft Azure > Azure Event Hub > Create a new data input, and then click the Next button.

    unnamed - 2025-02-19T115957.233.png

  3. If you don't yet have Azure Event Hubs enabled, click First time request guide and fill out the form that follows to request provision. This is usually provided within 24 hours. Otherwise, select Azure Event Hub and click Next.

    unnamed - 2025-02-19T120051.303.png

  4. Ensure Azure prerequisites are met, and then click Next.

    unnamed - 2025-02-19T120217.851.png

  5. Enter Azure Event Hubs data information. Note that Data Manager automatically validates the number of partitions. You will be able to change this in edit mode after data onboarding is complete. Click Review Data Input when you're ready.

    unnamed - 2025-02-19T120330.754.png

  6. Review the data inputs, then click Finish Setup and Monitor Configuration.

    unnamed - 2025-02-19T120415.552.png

  7. Monitor your inputs through Data Manager.

    unnamed - 2025-02-19T120419.987.png

Migrating from the Splunk Add-on for Microsoft Cloud Services to Data Manager

If you are migrating from the Splunk Add-on for Microsoft Cloud Services, follow these steps. Basic instructions for this process, without screenshots or additional guidance, can be found on Splunk Docs.

Prerequisites

  • You must be using at least version 5.4 of the Splunk Add-on for Microsoft Cloud Services.
  • Migration is supported only on Splunk Cloud Platform instances.
  • All prerequisites can be found in Data Manager under New Input → Microsoft Azure → Azure Event Hub → Migrate from an existing TA.

Step-by-step migration process

  1. Prepare for migration:
    1. Verify that your Event Hub inputs are set up and actively pulling data into the Splunk platform via the Splunk Add-on for Microsoft Cloud Services.
    2. Ensure that all inputs connected to the same Event Hub reside on the same Splunk instance.
  2. Check input health:
    1. Navigate to the Splunk Add-on for Microsoft Cloud Services.
    2. Select the Configuration tab.
    3. On the Export tab, set Input Status to an inactive state to prepare for export. This pauses ingestion at the current checkpoint. Data ingestion will resume from this checkpoint after migration.
    4. Ensure that the Ready for export option is checked.
    5. Click Export.

      unnamed - 2025-02-19T121300.629.png

  3. Export configuration snapshot:
    1. Ensure the Health Status is marked as Ready.
    2. Click Export to generate a JSON snapshot.
    3. The exported JSON includes:
      • Server information
      • Timestamp
      • Modular input configurations
      • Checkpoints (to prevent duplicate data ingestion)

        unnamed - 2025-02-19T121455.882.png

  4. Import JSON into Data Manager:
    1. Open Data Manager in Splunk Cloud Platform.
    2. Click the New Data Input button.
    3. Select Microsoft Azure → Azure Event Hub.
    4. Complete any remaining prerequisites.
    5. Upload the exported JSON file.
    6. Click Next.

      unnamed - 2025-02-19T123536.848.png

  5. Enter client secret:
    1. The client secret is not included in the JSON for security reasons, so you will need to enter it manually.
    2. Review the list of Event Hubs and associated inputs.
    3. Deselect any inputs you do not wish to migrate.
    4. Click Complete Migration of X Data Input to complete the migration process.

      unnamed - 2025-02-19T123637.907.png

  6. Verify post-migration data ingestion:
    1. Check that data is correctly ingested in Splunk Cloud Platform.
    2. Ensure that checkpoints were maintained to prevent duplicate data ingestion.

      unnamed - 2025-02-19T123656.075.png

  7. Cleanup:
    1. Wait 24 hours to confirm that all inputs are working correctly.
    2. Delete modular inputs from the Microsoft Cloud Services Add-on after verification.

Troubleshooting tips

  • Desynchronization issues: Ensure all modular inputs for the same Event Hub are on a single Splunk Cloud Platform instance.
  • Long processing times: Large instances with numerous modular inputs might experience delays during export/import.
  • Error messages: Errors related to index configurations or source types will be flagged during the migration process.

Next steps

These resources might help you understand and implement this guidance: