Skip to main content
Lantern Home
 
 
Splunk Lantern

Using ingest actions to filter Palo Alto logs

  1. Last updated
  2. Save as PDF

 

Palo Alto Networks (PAN) logs can pose challenges for Splunk platform users because of their volume and complexity. These logs, while rich in information, can inundate cloud environments, producing a lot of noise and leading to inefficiencies. This article introduces a strategic approach to filtering Palo Alto traffic and threat logs using ingest actions to ensure that only relevant data reaches your cloud environment, enhancing query efficiency and speeding up delivery of results.

Solution

You can manage Palo Alto logs more easily by focusing on data quality through targeted ingestion, ensuring that only relevant and clean data is processed. Ingest actions offers a sophisticated toolkit for refining data at the point of ingestion. This feature allows you to selectively process logs based on predefined criteria, significantly improving data quality and operational efficiency.

To set up ingest actions, you should follow the specific ingest actions requirements for your environment. See instructions for Splunk Enterprise or Splunk Cloud Platform. For comprehensive details and guidelines on the prerequisites and capabilities of ingest actions, see Ingest actions requirements.

Configuring ingest actions

To follow the steps below, you'll need to ensure the Palo Alto Networks Add-on for Splunk is installed and configured, as well as verify that you have access to ingest actions with the appropriate role permissions.

To get started, use to the following table to map the Common Information Model (CIM) to Palo Alto Networks event types:

CIM data model

Tags

Palo Alto Networks event types

Intrusion detection

ids, attack

pan_threat

Network traffic

network, communicate

pan_traffic

When filtering Palo Alto logs, you'll want to ensure that you filter low value data, keep high value data, and trim fields not required for troubleshooting.

For this solution we will focus on Palo Alto traffic and threat logs because those, on average, are the biggest source of PAN logs.

Fields to drop (pan_*)

You'll need to filter out events that have no defined value or when there is alternate field that should be used instead:

  • future_use_ - There is no defined usage for this field.
  • time - Redundant field. The true timestamp should use the field generated_time.

Fields to trim

(pan_threat)

The following pan_threat fields are not required for troubleshooting and can be trimmed. These fields do not align with the CIM definition for intrusion detection as specified by the Palo Alto add-on documentation for pan_threat event types.

  • *_dag
  • *_edl
  • app_*
  • cloud_report_id
  • container_id
  • dest_dvc_*
  • dest_location
  • dynusergroup_name
  • dvc_serial_number
  • future_use*
  • generated_time
  • high_res_timestamp
  • host_id
  • http2_connection
  • http_headers
  • justification
  • nssai_sst
  • partial_hash
  • payload_protocol_id
  • pod_*
  • reason
  • receive_time
  • rule_uuid
  • sctp_*
  • sequence_number
  • src_dvc_*
  • src_location
  • url_category_list
  • xff_ip

(pan_traffic)

The following pan_traffic fields are not required for troubleshooting and can be trimmed. These fields do not align with the CIM definition for intrusion detection as specified by the Palo Alto add-on documentation for pan_traffic event types.

  • *_dag
  • *_edl
  • app_*
  • container_id
  • dest_dvc_*
  • dvc_serial_number
  • dynusergroup_name
  • future_use*
  • generated_time
  • high_res_timestamp
  • host_id
  • http2_connection
  • link_*
  • nsdsai_*
  • offloaded
  • pod_*
  • policy_id
  • receive_time
  • rule_uuid
  • sctp_*
  • sdwan_*
  • session_owner
  • src_dvc_*
  • tunnel_start_time
  • xff_ip

Configuring ingest actions to drop fields

Follow these steps to create a ruleset:

  1. Navigate to Settings > Data > Ingest Actions.
  2. Provide a Ruleset Name to create a new ruleset. For example, PAN_DROP_FIELDS_FILTER_RULESET
  3. Identify and select the appropriate source for your logs, such as pan_*:*
  4. Select +Add Rule > Filter > Filter with regular expression to start the filtering process.
  5. Set Source Field as _raw.
  6. Set Drop Events Matching Regular Expression as future_use_ | time
  7. Click Apply to review the results.
  8. On the right hand side, ensure that the filter correctly filters the events. The events highlighted in red will be dropped before data is ingested.
  9. Click Save to save the new ruleset.

Configuring ingest actions to trim logs

Follow these steps to create a ruleset:

  1. Navigate to Settings > Data > Ingest Actions.
  2. Provide a Ruleset Name to create a new ruleset. For example, PAN_TRAFFIC_LOGS_TRIM_RULESET or PAN_THREAT_LOGS_TRIM_RULESET
  3. Select a data source type, for example, pan_traffic or pan_threat
  4. Select +Add Rule > Mask > Mask with Regex to start the filtering process.
  5. Set Match Regular Expression. See Gist for more information.
  6. In the Replace Expression field, enter a blank space.
  7. Click Apply to review the results.
  8. On the right hand side, ensure that the filter correctly trims the events. The events highlighted in red will be trimmed before being ingested.
  9. Click Save to save the new ruleset.

By following these steps, you've successfully configured ingest actions to filter out unnecessary fields in PAN threat and traffic, resulting in better data management.

Next steps

These resources might help you understand and implement this guidance:

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you would like assistance.