Skip to main content
Lantern Home
 
 
Splunk Lantern

Using ingest actions to filter AWS CloudTrail logs

  1. Last updated
  2. Save as PDF

 

AWS CloudTrail logs can pose challenges for Splunk Cloud Platform users because of their volume and complexity. These logs, while rich in information, can inundate cloud environments, producing a lot of noise and leading to inefficiencies. This article introduces a strategic approach to filter AWS CloudTrail logs using ingest actions to ensure that only relevant data reaches your cloud environment, which will enhance query efficiency and speed up delivery of results.

Ingest actions offer a sophisticated toolkit for refining data at the point of ingestion. This feature allows you to selectively process logs based on predefined criteria, significantly improving data quality and operational efficiency.

For comprehensive details and guidelines on the prerequisites and capabilities of ingest actions, see Ingest actions requirements.

To follow the steps below, you'll need to ensure the Splunk Add-on for Amazon Web Services is installed and configured, as well as verify that you have access to ingest actions with appropriate role permissions.

When filtering AWS CloudTrail logs, make sure you filter low-value data, keep high-value data, and trim fields not required for troubleshooting. Let’s walk through why you might want to filter or trim CloudTrail Logs first, followed by what logs you'll want to drop, and what fields you'll want to trim.

To see samples, go to: AWS CloudTrail Log file examples.

Drop logs

You'll filter your events by CloudTrail EventNames inside CloudTrail logs. You should filter the following API descriptors:

  • List. This response is overly verbose. The list is not critical in troubleshooting; instead, the count of List API Calls is more valuable.
  • Get. This response is overly verbose. The response is not critical in troubleshooting; instead, the count of Get API Calls is more valuable.
  • Describe. This response is overly verbose. The actual description of a service is not critical in troubleshooting.

Trim logs

The following fields are not required for troubleshooting and can be trimmed:

  • .responseElements.credentials.sessionToken - Temporary token generated for that session.
  • .requestParameters.resultToken - An encrypted token that associates an evaluation with an AWS Config rule.

These fields can likely be trimmed due to their low value, high cardinality, and because they do not assist with troubleshooting.

Configuring ingest actions to drop logs

For Splunk Cloud Platform, perform these steps to create a ruleset:

  1. Navigate to Settings > Data > Ingest Actions.
  2. Click to create a New Ruleset and provide a Ruleset Name. An example of this is:
    • AWS_CLOUDTRAIL_LOGS_GET_LIST_DESC_FILTER_RULESET
  3. Identify and select the appropriate data source for your logs, such as aws:cloudtrail.
  4. Select +Add Rule > Filter > Filter with regular expression to start the filtering process.
  5. Set Source Field as _raw.
  6. Set Drop Events Matching Regular Expression as List|Get|Describe.
  7. Click Apply.
  8. Ensure that the filter correctly filters the events. You can view this on the right hand side. The events highlighted in red will be dropped before data is ingested.
  9. Click Save to save the new Ruleset.

Configuring ingest actions to trim logs

For Splunk Cloud Platform, perform these steps to create a ruleset:

  1. Navigate to Settings > Data > Ingest Actions.
  2. Click to create a New Ruleset and provide a Ruleset Name. Two examples of this are:
    1. AWS_CLOUDTRAIL_LOGS_SESSIONTOKEN_TRIM_RULESET
    2. AWS_CLOUDTRAIL_LOGS_RESULTTOKEN_TRIM_RULESET
  3. Identify and select the appropriate data source for your logs, such as aws:cloudtrail.
  4. Select +Add Rule > Mask > Mask with Regex to start the filtering process.
  5. Set Match Regular Expression as:
    • .responseElements.credentials.sessionToken
    • .requestParameters.resultToken
  6. In Replace Expression enter a blank space.
  7. Click Apply.
  8. Ensure that the filter correctly trims the events. You can view this on the right hand side. The red highlighted events will be trimmed before being ingested.
  9. Click Save to save the new Ruleset.

You now have successfully configured ingest actions to filter out unnecessary verbosity from AWS CloudTrail Logs and streamline your data before it enters Splunk Cloud Platform.

Next steps

These resources might help you understand and implement this guidance:

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you require assistance.