Leveraging LLM reasoning and ML capabilities for Jira alert investigations
Security analysts often struggle to know where to begin an investigation in an ocean of alerts.
Correlating alerts or adding contextual information to data could make an analyst's job easier. When we apply this principle to machine learning (ML), we can build powerful models that are superior at detecting outliers. However, it can be difficult for someone unfamiliar with SPL or statistical analysis to know how to operationalize an ML model.
The Splunk Model Context Protocol (MCP) server for Splunk Cloud Platform works with LLMs and integrations like Jira to allow even novice Splunk users to utilize machine learning models without having to write the SPL queries needed to apply them. When multiple MCP servers like Jira work in conjunction, ML models become more impactful through properly associated data sources that create more robust models. Machine learning algorithms can analyze historical data trends to determine numeric and categorical outliers, allowing you to extract the required ML model to detect outliers in transactions and harness custom datasets from enriched data.
Data required
- Atlassian: Jira
- While this use case shows the use of financial data, you can adapt the steps shown in this process to prompt your LLM to deploy ML against any enriched data in the Splunk platform.
Scenario
A financial services company's security operations center (SOC) team monitors user behavior, transaction details, and merchant logs. Incidents generate alerts in Jira for analysts to investigate. A key part of their work involves analyzing bank transactions for suspicious activity, particularly when transactions involve multiple partner merchants.
To enhance this process, the team aims to:
- Correlate similar alerts to reduce noise and identify relevant data sources.
- Enrich this data to create more valuable datasets for machine learning (ML) inferencing.
- Implement anomaly detection models that analyze multiple data streams, including
authentication_logs,user_access_logs,merchant_transactions,bank_transactions, and business metrics like transaction volumes.
1. Add the Atlassian MCP to your LLM
If you haven't already, follow the configuration guidance in Leveraging Splunk MCP and AI for enhanced IT operations and security investigations. To add the Atlassian MCP when using a desktop LLM application that supports MCP, add the following JSON to your configuration file:
{
"mcpServers": {
"atlassian": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://mcp.atlassian.com/v1/sse"]
}
}
}
The first time your LLM calls an Atlassian tool, it will route you through a one-time authentication to ensure you have the correct permissions.
2. Correlate alerts and add context from Jira
First, we'll need to reduce the amount of alerts so we can decipher the necessary data sources for our analysis. If we can enrich data by understanding the contextual significance we can reduce scope and save time.
We provide this prompt:I am a security analyst who needs to investigate a potential security alert in Jira. I just received an "abnormal user behavior" alert. I believe there are other alerts that might be correlated but I'm not sure. Can you correlate these alerts together to look for a potential abnormal user behavior and give direction where to investigate?
The response received was:
Our LLM was able to determine that from the master alert there were several correlated alerts. From there, it was able to identify an investigative approach and relevant fields from the associated Splunk indexes listed on the Jira tickets themselves.
3. Deploying ML against enriched data within the Splunk platform
Now that we have an investigative plan, we can begin working on developing anomaly detection models that will be useful in determining potential signs of fraud.
We provide this prompt:
If there is abnormal user behavior then I want to ensure that no other strange activity is occurring. I want to apply machine learning within Splunk to detect anomalies across user behavior and transaction information, looking into merchants. If possible I'd like to leverage ML models that are already applied in saved searches. If you have questions or get confused, stop and ask me for guidance.
The response received from the LLM, confirming the plan, was:
The findings received were:
Here's a breakdown of the LLM's investigative steps:
Discovery of pre-existing models
The LLM leveraged purpose built models from saved searches in the Splunk environment to deploy against the custom dataset. After an initial investigation, the LLM found some statistical models that can be used to look for outliers.
Statistical analysis of transaction data (second hypothesis)
After not being able to find definitive outliers in user behavior from access and authentication logs, the LLM decided to look at transaction amounts for outliers:
index="transactions"
| rex field=_raw "(?<timestamp>[^,]+),(?<transaction_id>[^,]+),(?<user_id>[^,]+),(?<amount>[^,]+),(?<ip_address>[^,]+),(?<country>[^,]+),(?<status>[^,]+)"
| eventstats avg(amount) AS avg_amount, stdev(amount) AS stdev_amount BY user_id
| where amount > (avg_amount + (3 * stdev_amount))
| sort - amount
With this statistical approach, it identified a list of users with transactions that were more than three standard deviations above their personal average. This provides the first concrete lead.
Data enrichment and correlation (challenges encountered)
The LLM attempted to enrich this list of outlier users by correlating their activity with the merchant_transactions and bank_transactions indexes, but these correlation attempts failed. The primary reasons were data inconsistencies:
- The
bank_transactionsindex did not contain amerchant_namefield, making a direct join impossible. - The
user_idfield inbank_transactionswas a multi-value field, which complicated the use of subsearches.
Although we weren’t able to enrich our data in this way, we still signaled to our analyst the need to add certain elements together. These are common real-world data challenges.
Analysis of transaction failures (third hypothesis)
Given the data correlation issues, the LLM pivoted again. A new simplified hypothesis was formed: the abnormal behavior could be a high rate of failed transactions, which is a strong indicator of fraudulent activity.
This was the breakthrough. The query returned a large list of users with a 100% transaction failure rate. This signal was much stronger and more widespread than the high-value transaction outliers.
Final verification and reporting
To confirm the fraud hypothesis, the LLM cross-referenced a sample of these high-failure-rate users with the user_access_logs:
index="user_access_logs" (user_id="user_10681" OR ...)
The query revealed that the suspicious users had no login history, confirming they were likely not legitimate. This provided the final piece of evidence needed to conclude the investigation and generate actionable recommendations.
Next steps
After the LLM's initial investigation, it provided actionable recommendations. A key finding was the potential link between a repeatable error code and transaction fraud. You could use this information to create an alert within Splunk Cloud Platform to monitor similar events, search against a runbook in Confluence, or create a purpose-built ML model with the Machine Learning Toolkit (MLTK) to analyze this reduced dataset.
Additionally, the LLM's deeper analysis pinpointed three specific IP addresses (36.165.73.232, 41.8.18.214, 91.198.174.221) involved in both brute-force compromises and fraudulent transactions, confirming a single attacker. Based on this, the LLM automatically created a follow-up ticket to blacklist these IPs.