Skip to main content
Lantern Home

 

Splunk Lantern

Configuring native threat intelligence sources in Enterprise Security 8.5

  1. Last updated
  2. Save as PDF

Splunk Enterprise Security (ES) 8.5 ships with a powerful Threat Intelligence Framework (TIF) and prepackaged intel feeds you can use to build your defenses. However, threat intelligence isn't "plug and play". Threat intel sources downloaded to ES come in varied formats, and the raw data must undergo parsing and normalization before it becomes usable for detection and enrichment.

This article walks you through how ES native threat intelligence is processed behind the scenes, how to activate a prepackaged intel source, and how to add a custom feed of your own. After you have data flowing in, the companion article Detecting threats and enriching investigations with native threat intelligence picks up where this one leaves off, covering threat matching, alerts, and investigation observables.

Solution overview

The Threat Intelligence Framework moves data through five stages. The first two stages are covered in this article; the remaining stages are covered in the companion article on detection and enrichment.

  1. Adding threat intel sources — a threat list is downloaded from a configured source (prepackaged or custom) and parsed using source-specific directives.
  2. Verifying the download and routing — processed records are written to a dedicated KV store collection (one of nine *_intel collections), with routing determined by the fields the parser extracts.
  3. Seeing threat matching in action — granular threatintel_by_* collections derived from the *_intel collections are used by scheduled threat matching searches to surface IOC matches against your environment's events, with findings written to the threat_activity index when an IOC match is detected.
  4. Generating alerts from threat findings — threat detections such as Threat Activity Detected alert on findings, and the Threat findings dashboard surfaces them for analysts.
  5. Enriching investigations with threat intel observables — event-based detections can be edited to capture IOCs as observables, surfacing them in the Investigation Intelligence tab.

Adding threat intel sources

Threat intel sources come in two kinds: prepackaged sources that ship with ES, and custom feeds you configure yourself. Both flow into the same native KV store collections that the Threat Intelligence Framework manages. This section covers activating a prepackaged source, then adding a custom feed.

Activating a prepackaged threat intel source

Several prepackaged threat intel sources are enabled by default. Others, such as PhishTank (a list of community-verified phishing URLs), need to be activated manually.

To activate PhishTank, navigate to Configure > Threat intelligence > Data sources > phishtank and set it to Activated.

Phish_active.png

Because PhishTank is a prepackaged source, it has system-defined download and parsing directives. In the case of PhishTank, the parsing configuration is standard CSV pre-processing:

  • The Delimiting regular expression is a regex pattern that matches the comma delimiter.
  • The first line is a header and is skipped.

What needs special attention is the Fields expression. It maps url and description to columns in the source file using 1-based positional notation ($1, $2, etc.).

phish_config_UI.png

If you'd rather access the same configuration through the CLI, it lives in /SA-ThreatIntelligence/default/inputs.conf under the [threatlist://phishtank] stanza.

phish_stanza.png

You can download the original source file from the PhishTank URL to examine the schema. Note the url field mapped to the second column in the Fields expression; this is the essential data bucket of the dataset, storing the phish URLs. Based on the expression, you can also reconstruct what a sample record's description field looks like, for example: Target: Other (xref: http://www.phishtank.com/phish_detail.php?phish_id=...).

phish_list.png

If you need to add a custom feed, continue to the next section. Otherwise, skip ahead to Verifying the download and routing.

Adding a custom threat intel feed

While URL data from PhishTank is well suited to web proxy logs, another classic detection scenario worth examining is file hashes. To demonstrate how ES matches raw logs to intel, this example adds a custom feed, since you are not limited to prepackaged sources.

The file_intel collection in the example below is supplied with a batch of malicious file hashes submitted in the last 48 hours to Malware Bazaar, an open-source threat intel feed (a free account is required to generate an Auth-Key for the download URL).

To create a new, custom source feed with a recurring CSV download:

  1. Go to + Data source > Line Oriented.

    add_line_oriented.png

  2. Fill in the General details:
    • Name (becomes the threat_key).
    • Description.
    • Type – the recommended naming convention is threatlist_<intel_type>. The type is used to identify the threat category of IOCs in threat intel reporting.
    • URL - personalized with your Auth-Key, pointing to the source CSV.
    • Download interval – according to the feed's website, the minimum interval for this file shouldn't be shorter than 5 minutes. The TIF default of 43200 seconds (12 hours) is used in this example.

    malwarebazaar_add_1.png

  3. Fill in the Parsing details:
    • Delimiting regular expression
    • Skip header lines instructions
    • Fields mapping file_hash to the column storing the SHA256 hashes.

      Field mapping occurs after the delimiting regex is applied, so the column position might need to be shifted in advance in the mapping expression to account for any blanks the regex leaves behind.

      malwarebazaar_add_2.png

After the threat list is downloaded, parsed, and normalized, it contains file_hash with the downloaded SHA256 hashes alongside the already-introduced threat_key and time fields.

malwarebazaar_added.png

Verifying the download and routing

To confirm what happened after activating a data source, inspect threatlist.log by running the following search to search for PhishTank data. It's worth noting that this search is also a good starting point for troubleshooting issues with native intel downloads:

index=_internal sourcetype="threatintel:download" stanza=phishtank

The results lay out the PhishTank data retrieval workflow, with status messages walking you through each step chronologically:

  1. Threat list download.
  2. Parsing and normalizing.
  3. Writing the processed threat list to the appropriate KV store collection.
  4. Deleting the processed threat list file.

threatintel_download_statusmsg.png

The log event with status="Wrote records to collection" tells you exactly where the threat list was saved. The PhishTank intel ends up in a KV store collection called http_intel.

Why http_intel specifically? It's not random. There are nine dedicated *_intel collections in the Threat Intelligence Framework, and routing is determined by the fields the parser extracts. Because the PhishTank parsing configuration extracts a url field, that data is routed to the http_intel collection, which is designed to hold URLs. Similarly, if a feed extracts an ip and/or domain field, it's routed to the ip_intel collection. The custom Malware Bazaar feed, which extracts a file_hash field, is routed to the file_intel collection.

To inspect the collection, run:

| inputlookup http_intel

The results show the url and description fields extracted from the CSV, along with two new fields that were added automatically during normalization:

  • time – time of writing to the collection.
  • threat_key – name of the original source. This field allows you to easily attribute threat alerts back to this specific feed.

http_intel_phishtank.png

Don't be surprised if the number of records doesn't match the original file exactly. In the above test, the raw PhishTank CSV had 55,702 entries, of which 55,695 made it into the http_intel collection. A slight drop might occur because the header row is skipped and any malformed rows that fail the regex parsing rules might also be dropped.

Next steps

With both a prepackaged and a custom threat intel feed in place, the data is ready to be put to work. Detecting threats and enriching investigations with native threat intelligence covers what happens next: how threat matching searches surface IOC matches, how those findings become alerts, and how to enrich investigations with threat intel observables.

In addition, these resources might help you understand and implement this guidance: