Solarwinds alert management with ITSI

Configure Solarwinds alert actions (HTTP Post)

1. To edit or add an alert, click Add Action in the Trigger or Reset Action section of the Alert wizard.
2. Select the Send a GET or POST Request to a Web Server option, then click Configure Action.
3. Under HTTP request settings:
   1. Enter a URL in the field provided.
   2. Select either Use HTTP GET or Use HTTP POST.
   3. Enter the Body to POST.
      • Trigger

        {
        "source":"Solarwinds_Alerts",
        "event":{
        "timestamp": "${N=SWQL;M=SELECT GETUTCDATE() as a1 FROM Orion.Engines}",
        "vendor_severity": "${N=Alerting;M=Severity}",
        "severity_id":"${N=SWQL;M=SELECT TOP 1 CASE AlertConfigurations.Severity WHEN 0 THEN 1 WHEN 1 THEN 3 WHEN 2 THEN 4 WHEN 3 THEN 5 WHEN 4 THEN 6 ELSE 1 END AS ModifiedSeverity FROM Orion.AlertObjects INNER JOIN Orion.AlertConfigurations ON AlertObjects.AlertID = AlertConfigurations.AlertID WHERE AlertObjects.AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "app": "solarwinds",
        "description": "${N=Alerting;M=AlertMessage}",
        "signature":"${N=Alerting;M=AlertName}",
        "src": "${N=SWQL;M=SELECT TOP 1 RelatedNodeCaption FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "object": "${N=SWQL;M=SELECT TOP 1 EntityCaption FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "object_type": "${N=Alerting;M=ObjectType}",
        "itsiDrilldownURI": "${N=Alerting;M=AlertDetailsUrl}",
        "host_url": "${N=SWQL;M=SELECT TOP 1 RelatedNodeDetailsUrl FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "itsiDrilldownWeb":"Open Alert in Solarwinds",
        "solarwinds_object_id": "${N=Alerting;M=AlertObjectID}",
        "alert_id": "${N=Alerting;M=AlertDefID}",
        "sw_application": "${N=Generic;M=Application}",
        "ip": "${N=SwisEntity;M=IP_Address}",
        "nodename": "${NodeName}"
        }}

      • Reset

        {
        "source":"Solarwinds_Alerts",
         "event":{
        "timestamp": "${N=SWQL;M=SELECT GETUTCDATE() as a1 FROM Orion.Engines}",
        "vendor_severity": "reset",
        "severity_id":"2",
        "app": "solarwinds",
        "description": "${N=Alerting;M=AlertMessage}",
        "signature":"${N=Alerting;M=AlertName}",
        "src": "${N=SWQL;M=SELECT TOP 1 RelatedNodeCaption FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "object": "${N=SWQL;M=SELECT TOP 1 EntityCaption FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "object_type": "${N=Alerting;M=ObjectType}",
        "itsiDrilldownURI": "${N=Alerting;M=AlertDetailsUrl}",
        "host_url": "${N=SWQL;M=SELECT TOP 1 RelatedNodeDetailsUrl FROM Orion.AlertObjects WHERE AlertObjectID = ${N=Alerting;M=AlertObjectID} }",
        "itsiDrilldownWeb":"Open Alert in Solarwinds",
        "solarwinds_object_id": "${N=Alerting;M=AlertObjectID}",
        "alert_id": "${N=Alerting;M=AlertDefID}",
        "sw_application": "${N=Generic;M=Application}",
        "ip": "${N=SwisEntity;M=IP_Address}",
        "nodename": "${NodeName}"
        }}

   4. Starting with Orion Platform 2020.2, you can also specify the ContentType and Authentication:
      • None
      • Basic: enter or select credentials.
      • NTLM: enter or select credentials.
      • Token: enter a header name and value.
4. To schedule the action, select Time of Day > Use special Time of Day schedule for this action. This schedule only applies to the alert action you are editing. This is often used to prevent an action from occurring during specific windows.
5. Select how frequently this action occurs for each triggered alert in Execution Settings.
6. Click Add Action.

   Enter HTTPs instead of HTTP in the URL to send the information using HTTPS. The action is added to the trigger or reset action list, and you can test the action using the SIMULATE button. When the trigger or reset conditions of the alert are met, the GET or POST request is sent to the server. You can view the server logs to confirm that the action occurred.

   

Simulate a payload

The Solarwinds Configure Action activity has a SIMULATE button near the end of the configuration. This button can be used to simulate the payload. But keep in mind that none of the SWQL will be processed and will come across as clear text within the test payload. To see how the SWQL is processed the alert must be enabled and triggering on a condition.

During the ‘Simulate’ stage in Solarwinds you may encounter the following issue: Failed to execute HTTP request. This usually indicates problems with the authentication setup. Verify all information is built to the guide above.

Validate the data

At this stage, if everything is working correctly, you will see data flowing into the system. As you can see in the payload below, the line breaking isn't happening exactly where we want it, but all of the event data is making it into a single event (rather than multi-line).

In order to make the line breaking and field indexing work correctly, the appropriate source type needs to be used. This step is only needed if the source type is not already configured and installed as a part of the CPMA installation.

Additional troubleshooting steps for proper data ingestion might include installation of universal forwarder, configuration of DB Connect, installing TAs, or extracting KPI fields from log data.

Additionally, this will be a fairly new index, and as a result, you might need to use an all-time search to make sure timestamping is happening at the expected time.

Configure event analytics

In this stage, you will leverage the correlation searches and NEAPs provided by the Content Pack for Monitoring and Alerting to enable notable event and episode creation from Nagios alerts. You need to normalize the data according to the Alerts Data Model and configure the itsi_kpi_attributes and/or itsi_episode_contact_map lookups as appropriate to achieve the correct grouping of notable events and optionally configure which group to email when episodes are created.

Next, enable the Universal Correlation Search (UCS) according to the instructions found here and verify that it is working as expected. You can find more info on the capabilities of the UCS in Configuring the Universal Correlation Search to create notable events.

The Universal Correlation Search already includes noise-reduction methods (deduplication), but the performance of the correlation search can be improved by modifying the macro get_itsi_universal_index. The Universal Correlation Search (and certain drilldown searches) require a very broad ad-hoc search in order to find all normalized alerts. By default, this is index=*, which can be very expensive in some environments.

The index list can be modified via the macro get_itsi_universal_index. To improve search performance, provide a list of indexes, rather than index=*. To change the macro, perform the following steps:

1. Click Settings > Advanced Search > Search Macros.
2. Edit the macro get_itsi_universal_index. The default definition is index=* (index!=itsi_tracked_alerts AND index!=itsi_grouped_alerts).
3. Change the definition to include the list of indexes which include normalized alerts. For example: (index=nagios* OR index=solarwinds OR index=SplunkInfraMon) or ((index=alerts AND (sourcetype=nagios* OR sourcetype=solarwinds)) OR index=SplunkInfraMon)
4. Update the macro whenever new alert sources are added.

Next steps

If you have any trouble during this process, you should consider an engagement with Splunk Professional Services for further assistance. Click here to learn more about working with Professional Services.

Additionally, you can return to the main ITSI event management accelerator article for instruction on other third party alerts you can integrate into your deployment.