Enriching threat intelligence with cloud-based sources in Enterprise Security 8.5
In the first two articles of this series, we explored how to retrieve and operationalize threat intelligence from prepackaged and custom native threat intel sources. This article shows you how to take full advantage of Splunk Enterprise Security (ES) 8.5 threat intel enrichment with the cloud-based Splunk Threat Intelligence Management (TIM Cloud).
This article picks up from Detecting threats and enriching investigations with native threat intelligence. Because TIM Cloud is an ES integration that embeds into the native Threat Intelligence Framework, we recommend getting familiar with the first two articles to build a solid grasp of ES threat intel foundations before diving in here.
This article applies to Splunk Enterprise Security 8.5 Premier on-premises with Cloud Connect, or Splunk Cloud Platform with Enterprise Security 8. If you're brand new to TIM Cloud, we recommend starting with Overview of threat intelligence in Splunk Enterprise Security.
Solution overview
This article covers how TIM Cloud sources are accessed, stored, searched, and scored:
- Accessing and configuring TIM Cloud feeds — how TIM Cloud becomes available for on-premises ES deployments and what you need to activate cloud feeds.
- Understanding the TIM collection schema — how the
tim_iocscollection stores cloud intel and how it plugs into the native framework for threat matching. - Searching the TIM collection — how to surface the rich contextual data inside
indicator_jsonusingspath. - Interpreting threat scoring — how TIM Cloud normalizes source scores into the Priority Score shown in the Intelligence panel.
Accessing and configuring TIM Cloud feeds
With ES 8.5 Premier Edition, TIM becomes available for on-premises Splunk ES deployments through a Cloud Connect license.
To activate TIM Cloud feeds, you need to acquire, at a minimum, an API key that grants TIM access to download threat intel from your chosen API vendors. Some of these vendors are open source and let you either register an account to generate a limited-access key free of charge (for example, AbuseIPDB or AlienVault OTX) or connect to their feed with no explicit account or access key creation (for example, OpenPhish, Hybrid Analysis, or EU Cert).
Understanding the TIM collection schema
Let's start by looking at the TIM threat intel collection. In Search, run:
| inputlookup tim_iocs
In the screenshot below, the resulting table has been transposed and the indicator_json field truncated to fit all columns into view.
The tim_iocs collection holds all TIM Cloud intelligence, regardless of type. Each record represents an observable and has an assigned weight, download time, and threat_key, among others.
Let's pause on the threat_key. You already know that each native feed has its own threat_key that links it to observables. In tim_iocs, threat_key does not point to a source feed and shouldn't be interpreted that way.
What's good to know is that threat_key acts as a shared link between tim_iocs and the TIF *_intel collections.
To show this relationship, we'll search for one of the threat keys from tim_iocs in the corresponding *_intel collection. The key we've picked is assigned to an observable with observable_type=DOMAIN.
Based on what we've learned about the TIF *_intel collections, we should find this type of observable in ip_intel. Run | inputlookup ip_intel with the selected threat_key as a filter. The results are the IP and DOMAIN observables from tim_iocs that match the key, demonstrating that TIM Cloud intel is plugged into the existing framework architecture of KV store *_intel collections for threat matching.
Another important field you'll come across when browsing the collection is enclave_id. It's a unique identifier of the active threatlist that produced the observable. It's distinct from enclaveId within the indicator_json scoreContexts array (see the next screenshot), which identifies the source feed for the observable. There might therefore be more than one enclaveId attributed to an observable (in cases where at least two feeds report on one observable), but never more than one enclave_id.
Before we conclude the schema analysis, there's one thing worth pointing out about the indicator_json field. In our tim_iocs collection, we have a couple of IPv4 observables that were reported by more than one source feed, as evidenced by two enclaveNames (that is, source feed names) and enclaveIds recorded for these observables in the indicator_json:
This example demonstrates TIM collating intel on the observable_value from two disparate feeds, abuse_ipdb and eu_cert, before saving it to indicator_json. The indicator_json is therefore not the original payload (analogous to an event's _raw) from the feeds, but a product of TIM processing. The original feed payloads are saved in the cloud, and their processing—also run by a dedicated cloud service before the results reach your local Splunk instance—encompasses aggregation and normalization to the common JSON format you see in the collection.
Searching the TIM collection
You can reach the rich contextual observable data sitting inside indicator_json —such as the names of the sources or the IOC severity scores—and make it searchable at search time using the spath command. The following is only a sample to show you the breadth of information about TIM activity you can tease out with spath.
Let's see which sources are actively feeding observables into tim_iocs:
| inputlookup tim_iocs
| spath input=indicator_json output=threatlist_id path=workflowld
| spath input=indicator_json output=source_name path=scoreContexts{}.enclaveName
| spath input=indicator_json output=source_id path=scoreContexts{}.enclaveId
| mvexpand source_id | dedup source_id
| table threatlist_id, source_id, source_name
Notice there's just one threatlist_id across all results. That's because we've created and activated just one threatlist in our environment. Even if we had more, TIM Cloud allows no more than one to be active at a time. For reference, below is an example of tim_iocs in an environment where two threatlists were created and activated at some point; one was deactivated shortly before the other was activated.
Another handy search with spath lets you find out when each type of observable was most recently downloaded by each source feed. Since the source feed name isn't exposed as a standalone field, we use spath to extract it from the JSON.
| inputlookup tim_iocs
| spath input=indicator_json output=source_name path=scoreContexts{}.enclaveName
| eval downloaded_on = strftime(time, "%Y-%m-%d %H:%M:%S %Z")
| stats count(observable_value) AS "IOC Count", max(downloaded_on) AS "Most Recent Download", BY observable_type, source_name
| sort observable_type
Just one of many observations we can make about the results: DOMAIN observables are only supplied by the eu_cert feed in our environment, and they haven't been downloaded in the last few days. We can also see that none of our current feed subscriptions bring in certificate intel, which might (or might not) align with our threat matching and detection goals.
The observed variability in download dates might lead you to wonder about the download intervals configured for the feeds. This information isn't exposed in the UI because polling and download are fully automated in TIM Cloud with a common fixed interval of every 2 minutes. The reason you might not see equally recent data for each observable type is tied to the frequency with which the feed providers themselves refresh their threat intel databases, not the frequency with which TIM Cloud polls them.
Interpreting threat scoring
TIM Cloud uses a threat scoring method that retains the original source scores but also separately calculates a normalized score from all reporting sources. Score normalization is necessary because feeds that provide a score have their own scoring systems. For example, the Hybrid Analysis feed rates the severity of its IOCs with a discrete score between 0 and 100.
| inputlookup tim_iocs
| spath input=indicator_json path=priorityScore output=priorityScore
| spath input=indicator_json path=''scoreContexts{}.normalizedScore" output=normalizedScore
| spath input=indicator_json path="scoreContexts{).weight" output=weight
| spath input=indicator_json path="scoreContexts{}.sourceName” output=sourceNames
| spath input=indicator_json path="scoreContexts{}.passThroughScore.scoreValue" output=passThroughScore
| eval passThroughScore = tonumber(passThroughScore)
| stats
min(passThroughScore) AS min_raw_score
max(passThroughScore) AS max_raw_score
count AS ioc_count
BY sourceNames priorityScore normalizedScore
| eval score_range = min_raw_score . " - “ . max_raw_score
| fields - min_raw_score max_raw_score
| table sourceNames, priorityScore, normalizedScore, score_range, ioc_count
| sort sourceNames normalizedScore
TIM maps these original scores, stored as passThroughScore in indicator_json, to its own 4-step normalized scoring system. A normalizedScore of 0 indicates a BENIGN severity observable, 1 = LOW, 2 = MEDIUM, and 3 = HIGH. These verbally expressed severities constitute the final Priority Score that Enterprise Security displays for each TIM Cloud observable in the Intelligence panel of an investigation.
You can also find the original Pass through score assigned by each source that reported the observable in the same Intelligence panel, under Most recent reporting from cloud intelligence sources.
The weights applied to native observables don't count towards the normalized score calculated for observables cross-reported by TIM Cloud feeds. Currently, these scoring and weighing mechanisms don't overlap.
Next steps
This completes the series on working with threat intelligence in Splunk Enterprise Security 8.5, from native source configuration through detection, enrichment, and cloud-based intel with TIM Cloud.
In addition, these resources might help you understand and implement this guidance: