Hunting threats across Amazon S3-backed firewall logs using Federated Search in Splunk Cloud Platform

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

This article describes a pattern for using Splunk Cloud Platform Federated Search to search historical firewall telemetry stored in Amazon S3 and managed as Apache Iceberg tables through an Apache Iceberg REST catalog. The example focuses on Cisco Firewall telemetry for historical threat hunting, but the same architecture applies to other S3-backed datasets, including observability telemetry, application logs, compliance records, operational data, and financial datasets. In this pattern:

Amazon S3 remains the long-term storage layer.
Apache Iceberg provides table metadata and partition management.
Nessie provides the customer-managed catalog layer.
Apache Iceberg REST gives Splunk a standard interface to resolve table metadata.
Splunk Cloud Platform Federated Search provides an SPL2-based investigation experience without full reingestion into Splunk hot storage.

The core value is simple: keep historical data on Amazon S3, keep Splunk Cloud Platform as the investigation surface, and avoid reingesting the full dataset just to make it searchable.

Understanding why this matters

Security investigations rarely stop with recent data. A blocked connection this morning might need comparison with activity from last week, last month, or last quarter. A suspicious destination IP might require a historical lookback across firewall logs. High-severity intrusion events might need correlation with older connection activity, access control decisions, application behavior, or outbound traffic patterns.

Cisco Firewall telemetry from Cisco Secure Firewall Threat Defense, Cisco Secure Firewall Management Center, and existing Cisco ASA deployments helps you understand what traffic was allowed or blocked, which rule or access control policy matched, which application was involved, whether activity was tied to a known threat or intrusion signature, and whether there was unusual outbound movement by source, destination, application, or region.

A typical Cisco Firewall event can include fields such as event_time, event_date, action, src_ip, dest_ip, src_port, dest_port, protocol, app, user, security_zone, rule_name, access_control_policy, intrusion_policy, signature_id, threat_name, severity, bytes_in, bytes_out, packets, and region.

For recent and high-priority activity, this data is often searched directly in the Splunk Cloud Platform. But firewall telemetry grows quickly. Every allowed connection, blocked request, inspected flow, intrusion event, malware event, and policy decision can generate logs. Over time, keeping all historical firewall telemetry in hot Splunk storage might not be the right cost or operational model. Many teams retain older firewall data in Amazon S3, often in optimized formats such as Parquet, for long-term retention, audit, compliance, and historical threat hunting. The data remains available, but it might no longer be easy for you to search through your normal Splunk workflow.

Solving the business problem

After historical firewall telemetry moves outside Splunk indexes, you might need to understand S3 object paths, file layouts, schemas, partitions, table metadata, or separate query tools before you can answer basic investigation questions. During a security incident, that friction matters.

This proof of concept addresses that gap. It shows how historical Cisco Firewall telemetry can remain in Amazon S3, be managed as Apache Iceberg tables, and still be searchable from Splunk Cloud Platform using Federated Search.

How Splunk software helps with this use case

In this use case, you need to investigate historical firewall activity stored in Amazon S3. The data is managed as an Apache Iceberg table and exposed to Splunk Cloud Platform through an Iceberg REST catalog. The field values in the Cisco Firewall event data allow you to run a variety of investigation searches to narrow down the root cause of the incident.

You want to stay in Splunk Cloud Platform and answer questions such as:

Which high-severity blocked events occurred historically?
Which source IPs repeatedly attempted denied connections?
Which destination IPs or ports were associated with suspicious traffic?
Which firewall rules or intrusion signatures trigger most often?
Did outbound bytes spike for a specific application, region, or destination?
Was a known bad indicator seen in historical firewall telemetry?

With Federated Search, you can search the S3-backed Iceberg table from Splunk Cloud Platform without ingesting the full historical dataset into Splunk hot storage.

Designing the architecture pattern

This proof of concept uses Apache Iceberg REST because the scenario assumes you already manage, or want to manage, Amazon S3 data as Apache Iceberg tables. In this model:

Amazon S3 stores the physical Parquet data files.
Apache Iceberg tracks the table metadata.
Nessie provides the customer-managed catalog layer.
Splunk Cloud Platform connects to the Iceberg REST catalog to discover and query the table.
Splunk Data Management defines the federated dataset.
You query the dataset using SPL2.

Reviewing the architecture components

Layer	What it does	Why it matters
Splunk Cloud Platform	Runs SPL2 against a federated dataset backed by customer-managed Iceberg tables	You keep using Splunk Cloud Platform while historical firewall telemetry remains in Amazon S3.
Amazon S3 Iceberg warehouse	Stores Iceberg metadata, manifests, and Parquet data files	Amazon S3 remains the storage layer for historical telemetry while the data stays searchable from Splunk Cloud Platform.

Choosing Apache Iceberg REST

This pattern is not intended to imply that Iceberg REST is the only way to search Amazon S3-backed data. It is a reference pattern that is beneficial if you prefer an open table format and want explicit control over table metadata, schema evolution, snapshots, manifests, partition design, file locations, and catalog ownership.

A crawler-based approach can discover files and infer schema from data on Amazon S3, which can be useful for simple discovery workflows. However, for large or evolving datasets, you might need stronger table semantics and more predictable query behavior. The Apache Iceberg layer provides that table abstraction. In short, a crawler can help discover data, Iceberg defines and manages the table, and Iceberg REST gives Splunk Cloud Platform a standard catalog interface to that customer-managed table.

For historical Cisco Firewall telemetry, this matters because the dataset might grow over time, schemas might evolve, and partitions might be optimized by fields such as event_date, region, action, or severity.

Defining the example federated dataset

The proof of concept uses the federated dataset ~.federated.iceberg_firewall_logs_enriched.

The dataset is backed by an enriched firewall table in Amazon S3. It includes investigation-ready fields such as event time, action, severity, source and destination IPs, destination port, application, region, byte count, rule name, and signature ID. In your environment, the same pattern can point to a different Iceberg table and a different business, operational, or security dataset.

Connecting the investigation workflow

The SPL2 searches in this section are presented as one connected investigation path:

Confirm the S3-backed Iceberg dataset is searchable.
Prioritize high-severity blocked activity.
Roll up activity for monitoring and detection.
Narrow the investigation with partition-aware filters.
Triage known bad indicators and top talkers.

1. Confirm the S3-backed Iceberg dataset is searchable

The investigation starts with a validation step. Before analyzing risk, you confirm that the federated Cisco Firewall dataset is reachable from the Splunk Cloud Platform and that the expected fields are available. This step verifies the full access path: the Splunk Cloud Platform can reach the Iceberg REST catalog, the catalog can resolve the Iceberg table metadata, the S3-backed Parquet data is readable, and the federated dataset exposes investigation-ready firewall fields.

► Click to see the search, explanation, result, and analysis

Run the following search.

| from ~.federated.iceberg_firewall_logs_enriched
  | fields event_time, event_date, action, src_ip, dest_ip, app, region, bytes_in, bytes_out, rule_name, severity
  | head 100

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search	Explanation
`\| from ~.federated.iceberg_firewall_logs_enriched`	Query the federated dataset backed by the enriched Cisco Firewall table in Amazon S3.
`\| fields event_time, event_date, action, src_ip, dest_ip, app, region, bytes_in, bytes_out, rule_name, severity`	Return only the investigation-ready firewall fields to confirm they are populated.
`\| head 100`	Limit the result to a row-level sample of 100 events for a quick validation check.

Result

The result shows a row-level sample from the enriched Cisco Firewall table. You can see fields such as event_time, action, src_ip, dest_ip, app, region, byte counts, rule_name, and severity.

This gives you confidence that the historical firewall table is live and usable from the Splunk Cloud Platform. It also confirms that the data is not just technically reachable, but that the investigation is ready. Review the sample records and check that key fields are populated. If expected fields are missing, your next action is to validate the Iceberg table schema, source data mapping, or enrichment process.

2. Prioritize high-severity blocked activity

After confirming the dataset is searchable, you narrow the investigation to blocked traffic with high or critical severity. This turns a broad historical firewall dataset into a prioritized triage queue. Instead of scanning thousands or millions of firewall events, you focus on the blocked activity most likely to matter. In a Cisco Firewall context, this can help you identify:

Repeated deny events from the same internal source
Blocked traffic to suspicious destinations
High-severity intrusion or threat signatures
Rules that are repeatedly blocking risky behavior
Applications generating suspicious outbound attempts

► Click to see the search, explanation, result, and analysis

Run the following search.

| from ~.federated.iceberg_firewall_logs_enriched
  | where action="blocked" AND (severity="high" OR severity="critical")
  | stats count() AS blocked_events, sum(bytes_out) AS bytes_out BY severity, signature_id, rule_name, src_ip, dest_ip, dest_port, app
  | sort - blocked_events

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search	Explanation
`\| from ~.federated.iceberg_firewall_logs_enriched`	Query the federated Cisco Firewall dataset in Amazon S3.
`\| where action="blocked" AND (severity="high" OR severity="critical")`	Filter to blocked traffic with high or critical severity to build a prioritized triage queue.
`\| stats count() AS blocked_events, sum(bytes_out) AS bytes_out BY severity, signature_id, rule_name, src_ip, dest_ip, dest_port, app`	Group the events by severity, signature, rule, source IP, destination IP, destination port, and application, counting blocked events and summing outbound bytes.
`\| sort - blocked_events`	Sort the results so the highest-volume blocked combinations appear first.

Result

The result groups high and critical blocked events by severity, signature, rule, source IP, destination IP, destination port, and application. This allows you to see which combinations appear most often. For example, one source host might repeatedly hit the same blocked destination, or one firewall rule might be blocking repeated activity across multiple applications.

This provides fast prioritization. You do not need to manually inspect raw firewall events to find the highest-volume or highest-risk blocked activity. Start with the top rows by blocked_events and review which source IPs are repeatedly blocked, which destination IPs or ports are involved, which rule_name or signature_id triggered, whether the activity is isolated or widespread, and whether outbound bytes suggest attempted data movement.

3. Roll up activity for monitoring and detection

After you understand the highest-risk blocked activity, you can use the same historical table to create a broader operational context. This step summarizes Cisco Firewall activity by day, action, severity, and application. It helps you answer questions such as:

Is blocking traffic increasing over time?
Are high-severity events concentrated on specific days?
Which applications are associated with the most traffic?
Are there changes in allowed versus blocked activity?
Is there a spike that aligns with a known incident window?

► Click to see the search, explanation, result, and analysis

Run the following search.

| from ~.federated.iceberg_firewall_logs_enriched
  | stats count() AS events, sum(bytes_in) AS bytes_in, sum(bytes_out) AS bytes_out, sum(packets) AS packets BY event_date, action, severity, app
  | sort event_date, action, severity

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search	Explanation
`\| from ~.federated.iceberg_firewall_logs_enriched`	Query the federated Cisco Firewall dataset in Amazon S3.
`\| stats count() AS events, sum(bytes_in) AS bytes_in, sum(bytes_out) AS bytes_out, sum(packets) AS packets BY event_date, action, severity, app`	Roll up firewall activity into daily metrics for event volume, inbound bytes, outbound bytes, and packet counts, grouped by date, action, severity, and application.
`\| sort event_date, action, severity`	Order the results by date, action, and severity for trend analysis.

Result

The result rolls up firewall activity into daily metrics. It shows event volume, inbound bytes, outbound bytes, and packet counts by event date, action, severity, and application.

This turns historical Amazon S3-backed firewall telemetry into an operational dataset for trend analysis, recurring monitoring, scheduled searches, dashboards, or downstream summaries. The important point is that the historical data does not need to be fully reingested into Splunk Cloud Platform hot storage before it can support these workflows. Review daily trends and look for unusual spikes in blocked traffic, high outbound byte volume, applications with unexpected activity, days where high or critical severity events increased, and patterns that you should convert into a scheduled search or dashboard panel.

4. Narrow the investigation with partition-aware filters

After identifying the most relevant time window or region, you narrow the search scope using fields that align with the Iceberg table design. In this example, the table can be filtered by event_date and region. These are useful investigation fields, but they can also be important physical design fields when the data is partitioned in Amazon S3. This step demonstrates why table design matters. If the Iceberg table is organized around fields such as date and region, the Splunk Cloud Platform can use those filters to reduce the amount of historical data that needs to be scanned.

► Click to see the search, explanation, result, and analysis

Run the following search.

| from ~.federated.iceberg_firewall_logs_enriched
  | where event_date="2026 05 13" AND region="us east 1"
  | stats count() AS events BY event_date, region, app, action, severity
  | sort -events

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search	Explanation
`\| from ~.federated.iceberg_firewall_logs_enriched`	Query the federated Cisco Firewall dataset in Amazon S3.
`\| where event_date="2026 05 13" AND region="us east 1"`	Narrow the dataset to a specific date and AWS region, using partition-aware fields to reduce the data scanned.
`\| stats count() AS events BY event_date, region, app, action, severity`	Summarize events by date, region, application, action, and severity.
`\| sort -events`	Sort the results so the highest-volume activity appears first.

Result

The result narrows the dataset to a specific date and AWS region, then summarizes events by application, action, and severity.

This step shows the practical value of partition-aware design for large historical datasets. You can focus the investigation without needing to know Amazon S3 object paths, file locations, or Iceberg manifest details. Use this narrowed view to determine which applications were active in the target region, whether blocked or high-severity activity was concentrated on that date, whether the activity matches the incident window, and whether your next search should focus on a specific app, source, destination, or rule.

5. Triage known bad indicators and top talkers

The workflow closes with a focused triage search for known bad indicators and repeated activity. In this scenario, you look for events tied to a known bad IP indicator or a rule that denies known bad destinations. This is where enriched fields such as signature_id, rule_name, severity, src_ip, dest_ip, dest_port, and app become especially valuable. The goal is to identify who attempted the connection, where they were going, which application was involved, and how often it happened.

► Click to see the search, explanation, result, and analysis

Run the following search.

| from ~.federated.iceberg_firewall_logs_enriched
  | where signature_id="IOC BAD IP" OR rule_name="deny known bad ip"
  | stats count() AS attempts, sum(bytes_out) AS bytes_out BY src_ip, dest_ip, dest_port, app, severity, rule_name
  | sort -attempts

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search	Explanation
`\| from ~.federated.iceberg_firewall_logs_enriched`	Query the federated Cisco Firewall dataset in Amazon S3.
`\| where signature_id="IOC BAD IP" OR rule_name="deny known bad ip"`	Filter to events tied to a known bad IP indicator or a rule that denies known bad destinations.
`\| stats count() AS attempts, sum(bytes_out) AS bytes_out BY src_ip, dest_ip, dest_port, app, severity, rule_name`	Group repeated attempts by source IP, destination IP, destination port, application, severity, and rule name, counting attempts and summing outbound bytes.
`\| sort -attempts`	Sort the results so the top talkers appear first.

Result

The result highlights repeated attempts involving known bad indicators. It groups activity by source IP, destination IP, destination port, application, severity, and rule name.

This gives you a compact triage view. Instead of reviewing raw events one by one, you can immediately see the top talkers, repeated destinations, and rules associated with known bad activity. Use this output to decide next steps: investigate the top source IPs, review destination reputation and ownership, check whether traffic was blocked or partially allowed, correlate source hosts with endpoint, identity, or asset data, escalate repeated attempts for incident response, and create a detection or scheduled search for future recurrence.

Knowing when to use this pattern and what it delivers

This proof of concept validates a practical federated search pattern for Amazon S3-backed data managed through Apache Iceberg REST. It shows that Amazon S3 can remain the long-term system of record for historical firewall telemetry, Splunk Cloud Platform can remain your search and investigation surface, Apache Iceberg provides the table abstraction for schema, partitions, snapshots, manifests, and metadata, and Apache Iceberg REST gives Splunk Cloud Platform a standard way to resolve customer-managed table metadata. You do not need to reason about raw Amazon S3 object paths or file layouts, partition-aware fields such as event_date, region, app, and severity become practical search controls, and historical data does not need to be fully reingested into Splunk Cloud Platform hot storage to support investigation, audit, compliance, and threat hunting workflows.

This pattern is a good fit when you retain historical data in Amazon S3, want to manage that data as Apache Iceberg tables, need to search the data from Splunk Cloud Platform, do not want full reingestion into Splunk Cloud Platform hot storage, benefit from schema, partition, and snapshot control, and use the data for long-lookback investigation, compliance, audit, or reporting. It is especially useful for high-volume datasets such as firewall telemetry, where older data is valuable for investigation but might not need to remain in hot storage.

Next steps

With this approach, you can search Amazon S3-backed data directly from Splunk Cloud Platform, keep historical data in cost-effective object storage, avoid reingesting full datasets just to investigate them, use familiar SPL2 workflows across indexed and federated data, support longer retention, compliance, and historical threat hunting, and keep control of your customer-managed Apache Iceberg tables and metadata.

In addition, these resources might help you understand and implement this guidance:

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their Success Plan. Engage the ODS team at ondemand@cisco.com if you would like assistance.