Leveraging MLTK's new generative AI capability in security operations

Setup for the MLTK AI command

To get started, you’ll first need to configure the LLM connectors within MLTK.

1. Ensure that you have both MLTK 5.6.0 and Python for Scientific Computing 4.2.3 installed.
2. Verify that your user role includes the necessary capabilities for configuring and using the AI command. For more information, see About the ai command.
3. Navigate to the Connection Management tab in MLTK and select the LLM services you wish to set up.

For our demonstration, we configured both an on-premises Ollama LLM and a cloud-based Azure OpenAI LLM. The on-premises LLM is used for personally identifiable information (PII) data identification and finding analysis to ensure sensitive information remains within your environment, while the cloud-based LLM handles detection suggestions based on the MITRE ATT&CK framework, as this use case involves only generic data.

Use case 1: PII data identification

PII is often hidden within the vast amounts of data captured in security logs. Because PII can appear in unexpected fields or formats, manual review is time-consuming and prone to oversight. Automated methods, such as using LLMs, are increasingly essential for efficiently identifying and safeguarding sensitive information that might be buried deep within log records.

In this example, we examine sample application logs containing multiple fields, as illustrated in the figure below. PII might appear in fields such as ContactID and CommNumber, alongside non-PII fields like Status and TransactionID.

A key challenge in PII identification is that field names alone do not always clearly indicate the presence of sensitive information. As a result, it is often necessary to inspect field values directly - a process that can be tedious and error-prone if done manually. To address this, we use the following SPL to prompt the LLM for automated PII identification.

source="sample_application_log"
| foreach * [ eval k="<<FIELD>>", v='<<FIELD>>' | eval all_fields=mvappend(all_fields, k."=".v) ]
| mvexpand all_fields
| rex field=all_fields "(?<field>[^=]+)=(?<value>.*)"
| stats values(value) AS examples BY field
| eval examples=mvindex(examples, 0, 2)
| ai prompt = "Based on fieldname: {field} and its sample values {examples}, determine whether it contains PII information and explain why. Return strictly in this format <yes|no>,<reason>"
| rex field=ai_result_1 "(?<decision>\b(?:yes|no|YES|NO|Yes|No)\b|\[(?:yes|no|YES|NO|Yes|No)\]|\<(?:yes|no|YES|NO|Yes|No)\>)\s*[, ]+\s*(?<reason>.+)"
| eval decision=replace(decision, "[\[\]<>]", "")
| table field examples decision reason ai_result_1

In this SPL, we first extract each field name from the logs along with three example field values. These field names and sampled values are then included in the prompt within the ai command, allowing the local LLM to determine whether a field contains PII. By providing both the field name and representative values, the LLM can make more informed decisions. The number of sample values can be easily adjusted within the SPL to suit different use cases.

Finally, we use the rex command to extract the LLM’s decision and reasoning from the response field (ai_result_1), enabling us to identify which fields contain PII. The results of this SPL execution are shown in the image below.

The prompt can be adjusted flexibly to identify PII fields depending on the company guideline. Furthermore, adding examples in the prompt can also improve the accuracy.

Use case 2: Detection suggestions based on the MITRE ATT&CK framework

Developing effective detection rules can be a challenging task, as choosing the most relevant ones requires a deep understanding of both the latest attack techniques and the organization’s specific industry context. Since adversary tactics and techniques often vary by sector, it’s important to identify which MITRE ATT&CK techniques are most applicable to your environment. After you determine these techniques, the next step is to map them to corresponding Splunk detection rules to ensure that your security monitoring is both comprehensive and tailored to current threats.

Recent advancements in LLMs have significantly improved their understanding of the MITRE ATT&CK framework, as well as their awareness of trending attack techniques targeting specific industries. By leveraging these models, security teams can efficiently identify the most relevant techniques for their organization’s threat landscape. This makes LLMs a valuable tool for selecting which techniques to detect, ensuring that detection efforts are both up-to-date and industry-specific.

In this example, we leverage both the power of LLMs and the Splunk Security Essentials (SSE) framework to identify the most suitable detection content. The SPL query used for this process is provided below.

| makeresults | eval company_name = "Cisco" | eval num_results = "4"
| ai prompt="What is {company_name} and which industry they are in?" 
| ai prompt= "Are OT and IT usecases relevant to {ai_result_1}?" 
| ai prompt="Based on this organization: {ai_result_1} and attack surface relevancy: {ai_result_2}, suggest the top {num_results} MITRE attack techniques relevant to this organization, ONLY return the list of techniques ID separated by comma."
| rex max_match=0 field=ai_result_3 "(?<technique>T\d{4})"
| mvexpand technique
| dedup technique
| map maxsearches=100 search="
    | inputlookup mitre_detections 
    | eval mitre_id=\"$technique$\"
    | eval TechniquePrefix=mvindex(split(TechniqueIdCombined, \".\"), 0)
    | where TechniquePrefix=mitre_id
    | lookup mitre_data_sources Data_Component OUTPUT Description Data_Source
    | eval Data_Sources = \"** \" + Data_Source + \" **\" 
    | stats values(Data_Sources) as Data_Sources
    | eval technique=\"$technique$\"
    | eval ai_result_1=\"$ai_result_1$\"
    | eval ai_result_2=\"$ai_result_2$\"
    | eval ai_result_3=\"$ai_result_3$\"
"
| map maxsearches=100 search="
    | sseanalytics include_json=true
    | search mitre_technique=*$technique$*
    | eval Rules_Combined = \"** \" + name + \" **\"
    | stats values(Rules_Combined) AS Detection_Rules
    | eval technique=\"$technique$\"
    | eval ai_result_1=\"$ai_result_1$\"
    | eval ai_result_2=\"$ai_result_2$\"
    | eval ai_result_3=\"$ai_result_3$\"
    | eval Data_Sources=\"$Data_Sources$\"
"
| rename technique AS Related_Mitre_Technique, ai_result_1 AS Company_Info, ai_result_2 AS Company_Usecase
| ai prompt="Based on this Mitre attack technique: {Related_Mitre_Technique}, explain it in layman's term"
| rename ai_result_1 AS Technique_Explained
| table Related_Mitre_Technique Technique_Explained Data_Sources Detection_Rules

At the start of this SPL query, we specify the company name—which can be customized for any organization—and set the maximum number of MITRE ATT&CK techniques to output (set to 4 in this example). Then, we prompt the LLM to generate a description of the company and its industry based on the provided name. The output from this initial prompt is then used in a second LLM prompt, where we ask the model to generate relevant OT and IT use cases for the company and its industry. In the final step, we prompt the LLM again to identify a list of attack techniques targeting those specific use cases. By chaining these LLM prompts together, we enhance the model’s reasoning process, which leads to more accurate and context-aware results.

While we specify the output format in the final prompt, model responses can still vary. To address this, we use the rex command to extract technique codes from the LLM’s response to the third prompt. Each technique code is then processed through two subsearches: one retrieves the relevant data sources using SSE lookups, and the other identifies available detection rules for each technique. Finally, we prompt the LLM to provide a description of each technique, helping us gain a clearer understanding of the associated threats.

The output of this SPL query is provided in the following image.

For each technique suggested by the LLM, a corresponding set of data sources and detection rules is provided. Users can review these results in the context of their own environment to identify additional data sources to onboard or detection rules to enable.

This SPL query serves as an example for this use case. The prompts can be easily customized to focus on specific services or software types so you can discover the most relevant detection content for your needs.

Use case 3: Finding analysis on Splunk Enterprise Security

In Splunk Enterprise Security, findings are generated from the enabled detection rules and the ingested data. While some findings result from a single rule firing—leading to a standalone alert—others are derived from advanced mechanisms such as cumulative entity risk scoring and frameworks like MITRE ATT&CK or the Cyber Kill Chain. These advanced findings are designed to correlate multiple related alerts, helping analysts identify complex attack techniques or campaigns that span several stages.

Cumulative findings provide valuable context by aggregating related events, allowing analysts to better understand the progression of an attack and the entities affected. However, to gain a comprehensive view of an incident, analysts often need to review each individual alert, which can be both time-consuming and susceptible to human error.

The following image illustrates a finding based on the detection of multiple MITRE ATT&CK tactics observed over a seven-day period. This finding aggregates 24 individual alerts. In this example, we explore how LLMs can be leveraged to analyze a sequence of alerts using the MLTK ai command.

One challenge when using LLMs to analyze a large volume of data at once is accommodating the model’s context window limitations, particularly with local LLM deployments.

In this example, we address this by using the following SPL command to:

1. Retrieve the raw events associated with the risk object (user "fyodor") from the Risk data model in Splunk Enterprise Security.
2. Obtain the detection description for each alerting event.
3. Extract the timestamp, detection description, risk object, and risk score for each individual alert.

By summarizing the alert data in this way, we reduce the volume of information to fit within the LLM’s context window.

Finally, we use the ai command to prompt the model, enabling it to generate a structured analysis of the finding.

| from datamodel:"Risk.All_Risk" 
| normalizeip normalized_risk_object 
| search normalized_risk_object="fyodor" risk_object_type="user" annotations.mitre_attack.mitre_tactic_id=* | `get_correlations` 
| rename annotations.mitre_attack.mitre_tactic_id as mitre_tactic_id, annotations.mitre_attack.mitre_tactic as mitre_tactic, annotations.mitre_attack.mitre_technique_id as mitre_technique_id, annotations.mitre_attack.mitre_technique as mitre_technique 
| noop search_optimization.search_expansion=false
| table _time, rule_name, risk_object, risk_message, risk_score, mitre_technique_id 
| sort _time
| eval details = "Time=" + _time + ", Rule Name=" + rule_name + ", Risk Score=" + risk_score + ", Risk Message=" + risk_message 
| stats values(details) BY risk_object
| ai prompt=" Based on respective alerts, where each alert starts with "Time=", and their rule name, risk score and risk message in {values(details)}, where the time value is a epoch time value, provide a concise summary of the security incident which has happened to {risk_object} in chronological order, create a timeline on what happened at what time (in the format of YYYY-MM-DD HH:MM:SS) with details at each time and the potential impacts. Provide an analysis on how the risk score accumulated to a final score based on the behaviors of {risk_object}. Provide an overall analysis of this finding and suggestions on investigation and triaging."

The output of the model is shown below. The LLM produced a clear timeline of detected events along with their associated risk scores. Additionally, it summarized the key findings in a human-readable format and provided actionable suggestions for further investigation. With this analysis, users can easily interpret the alerts and determine appropriate next steps.

Since the LLM analysis is implemented via SPL, it can also be registered as a drill-down search within the Splunk Enterprise Security framework. This enables seamless execution of the analysis directly from a finding, enhancing the efficiency and effectiveness of security investigations.