Getting started with Microsoft Azure Event Hub data

Overview

There are four basic steps to follow:

1. Creating an Event Hub Namespace
2. Sending Data to an Event Hub
3. Setting up Authentication
4. Setting up the Splunk Add-on for Microsoft Cloud Services

If you already have data in an Event Hub you want to ingest, skip to step 3 to ensure authentication is set up correctly for the Splunk add-on.

Step 1 - Create an Event Hub Namespace

An Event Hub Namespace is a container for individual Event Hubs. You aren’t creating any Event Hubs in this step, just creating a home for Event Hubs.

1. Log into the Azure portal - https://portal.azure.com
2. Type event hub in the search bar and click the Event Hubs service.

3. Click the Create button.

4. Fill out the details and click the Create button.

Step 2 - Send data to an Event Hub

Use an Azure Monitor diagnostic setting to send Azure activity logs to an Event Hub. A diagnostic setting defines what to send where. In this case, you are going to send Azure Activity logs to an Event Hub. This same technique of creating a diagnostic setting can be used for most services in Azure as well as Azure Active Directory.

1. Type monitor in the search bar and click the Monitor service.

2. Click Activity log.

3. Click the Diagnostics settings button.

4. Click Add diagnostic setting.

5. Give the diagnostic setting a name, select which log categories you want to export to the Event Hub in the Categories section, and select Stream to an event hub in the Destination section. When selecting the Stream to an event hub option, select the Event Hub Namespace.

 

6. Click Save.

Step 3 - Set up authentication

In order to consume data from an Event Hub, some sort of authentication needs to happen. This is typically done via a connection string or an Azure AD application registration. The Splunk Add-on for Microsoft Cloud Services uses an Azure AD application registration, so that is what you'll cover next.

1. Type azure active directory in the search bar and click the Azure Active Directory service.

2. Select App registrations.

3. Click the New registration button.

4. Give the application registration a name, choose Single-tenant as the account type, and click the Register button. The Redirect URI is not needed.

5. After the application is registered, an overview screen will appear. Copy the Application (client) ID and Directory (tenant) ID. You'll need to use these in step 5.

 

6. Click Certificates & secrets as shown in the screenshot in step 5, above. Then click the New client secret button.

7. Provide a description and expiration date, and then click the Add button.

8. After the secret is created, the value will be displayed in the Azure portal. The Secret ID is not used. This is the only time you will see the value in plain text, so make a copy of it now as you will need it later. If you lose this value, you will need to create a new secret.

Step 4 - Set up role assignments

Now that you have your Azure AD application registration, you'll need to add it to the Azure Event Hubs Data Receiver Role.

1. Type subscriptions in the search bar and click the Subscriptions service.

   

2. Select your subscription, and then click Access control (IAM).
3. Click the Add button, and then select the Add role assignment option.

   

4. In the search box, type Azure Event Hubs. Select the Azure Event Hubs Data Receiver role in the table, and then click the Next button.

   

5. Click the Select members button. Type the name of the Azure AD Application Registration you created earlier. Click the name of the application so that it shows up in the Selected members section. Then, click the Select button.

   

6. Click the Review + assign button.
7. Once the process finishes, click Role assignments and check that the application is in the Azure Event Hubs Data Receiver role.

   

   Repeat step 4 for any other subscriptions that contain Event Hubs that will be consumed by Splunk.

Step 5 - Configure the Splunk Add-on for Microsoft Cloud Services

Now, configure the Splunk Add-on for Microsoft Cloud Services.

1. Under the Configuration menu, ensure Azure App Account is selected, and then click the Add button.

   

2. Give the account a name and paste the values obtained in step 3. Then, click the Add button.

   Client ID = Application (client) ID
   
   
   Key (Client Secret) = the value of the client secret
   
   
   Tenant ID = Directory (tenant) ID

   

3. Click the Inputs menu item, then click the Create New Input button and select Azure Event Hub.

   

4. Provide a name for the input and select the account created in the previous step.

   

5. To get the value you'll need to enter into the Event Hub Namespace (FQDN) field, go to the Azure portal again. Type event hub in the search bar and click the Event Hubs service. A list of Event Hub Namespaces will be displayed.

   

6. Select the appropriate Event Hub Namespace in the table, and an overview page should appear. The Host name value is the Event Hub Namespace (FQDN). In the below image, the Event Hub Namespace (FQDN) is splunkblog.servicebus.windows.net .

   

7. An Event Hub Namespace contains one or more Event Hubs. To get the Event Hub name, first click on Event Hubs under the Entities section from the same screen as step 6, above. You may need to scroll down to see this. You'll then see your individual Event Hubs displayed in a table. If you let Azure create an Event Hub for you when setting up your Activity log diagnostic setting earlier, you'll see the Event Hub name that it created. In this example the Event Hub name is insights-activity-logs. This is the value you need for the Splunk input.

   

8. Your final input should look like the following:

   

   Add in the following settings:

   • Consumer group Every application that reads data from an Event Hub must have a distinct consumer group. For example, if both Splunk and a custom application are both reading from the same Event Hub, then each would need a unique consumer group. Consumer groups allow multiple consuming applications to each have a separate view of the event stream.
   • Transport Type. This is the outbound protocol that the Splunk forwarder will use to connect to the Event Hub. AMQP over WebSocket (the default) will use TCP 443 (https). If you select AMQP, then port 5671 outbound needs to be open. If you use AMQP with a proxy, port 5672 outbound needs to be open.
   • Sourcetype. This is the Splunk sourcetype that will get tagged with the data (mscs:azure:eventhub is the default). Typically, the same type of data is in the same Event Hub. It's recommended to set a sourcetype value for the type of data you are consuming from the Event Hub, instead of using the default. This will make searching easier down the road.
9. Click the Add button, and you should start seeing data in your index.

   

   Repeat the procedures in step 5 for any additional event hubs you want to consume.