Integrating Splunk ITSI with Splunk AppDynamics for unified business performance monitoring

Step 1: Installing and configuring the Content Pack for Splunk AppDynamics and the Splunk Add-on for AppDynamics

The Content Pack for AppDynamics and Cisco Splunk Add-on for AppDynamics are both required to integrate Splunk ITSI with AppDynamics monitorable entities. You will be able to configure the specific data inputs and monitorable entities that you want to integrate into ITSI.

1. Download and install the Content Pack for AppDynamics, accessible through the Splunk App for Content Packs, and follow the instructions in the documentation to configure it.
2. Download and install the Cisco Splunk Add-on for AppDynamics on your Splunk platform instance and follow the instructions in the documentation to configure it to integrate with your AppDynamics instance.
3. Follow the instructions in the documentation to help you select the data inputs you want to collect. Inputs you might want to add include:
   • Health rule violations
   • Business transaction metrics
   • Application performance data

Step 2: Send AppDynamics alerts to ITSI

In ITSI, navigate to Configuration > Data integrations > Integrations library > AppDynamics to configure HTTP request webhooks so that alerts from AppDynamics flow into ITSI.

Step 3: Review related events in ITSI and identify root cause within an episode

When issues occur, ITSI creates episodes that group related alerts together. Instead of receiving ten or twenty alerts when a nonstandard pattern occurs, you receive one episode with all related information from all the entities monitored by ITSI, including AppDynamics and other systems.

1. From the Service Analyzer, navigate to the ITSI Episode Review dashboard and click View All.
2. Review episodes that have been created for emerging issues and click into an episode for further drilldown.
3. Examine the event timeline to see:
   • Which KPIs and tests have failed
   • When they transitioned between states
   • The sequence of events leading to the issue
4. In the Events Timeline tab, click to sort by root cause analysis to identify the most likely source of the problem.

The screenshots below show you an alert group indicating issues with the health of an AppDynamics business transaction, along with a timeline where you can see how these issues occurred sequentially.

Step 4: Deepen your investigation using service metrics from AppDynamics

After you get a health alert, you can see what's happening by looking at service metrics from AppDynamics to see where to go next.

1. From the Service Analyzer, review the service topology to identify potential issues by severity.
2. In the KPIs and Episodes pane, review the relevant KPIs which leverage AppDynamics metrics such as application health, application response time, or error percent.
3. Click on a KPI to drill deeper on the entity. The screenshot below shows a drilldown into an e-commerce application that is causing the Application Health KPI to go to critical severity. The entity view provides detailed metrics for the e-commerce application, including response time, error percent, and more. 

Step 5: Dive into root cause with AppDynamics using deep linking in ITSI

After you have identified metrics related to the issue, such as high error percent or high latency, you can drill directly into AppDynamics to the related entity.

1. From the previous screen, click the tree icon at upper right to slide out the Entity Information view.
2. Click Drilldown in AppDynamics to be taken directly to the entity, time range, and view relevant to the issue.

Step 6: Understand root cause using deep dive tools in AppDynamics

From the dashboard in AppDynamics, you can see which microservice the issue is coming from and that, in the example screenshot shown below, several measurements of slow response time triggered the violation.

1. From the previous view, you will already be in the transaction flow map during the timeframe when the issue occurred.
2. In the right navigation pane, select Transaction Scorecard. Click Very Slow to view the transactions captured in the very slow category and double-click to deep dive into one of these captured transactions with a high end-to-end latency time.
3. In the Transaction map, you will see how many milliseconds were spent in each service and potential issues identified from anomalies identified in the transaction in the Summary view. You will also see visual indicators (red, orange, yellow, and green) indicating performance and you can drill into these to investigate them in a call graph. Click Drill Down to investigate code performance at the service level.
4. Inside the call graph you will see the underlying code performance breakdown where you can identify root cause, including the exact method within the code and the outbound calls to identify the performance bottleneck.
5. From the Transaction map or the Transaction screen you can also drill down further to view application logs related to the transactions, which provide greater detail and specificity on the cause of performance issues or errors.

You have now found the root cause and can work with your application team to fix the application quickly and efficiently.