Skip to main content
Lantern Home
 
 
Splunk Lantern

Reducing low-value data ingestion to improve license usage

  1. Last updated
  2. Save as PDF

 

Data that is weeks and months old is ineffective for real-time use cases but remains important for customers with compliance, auditing, and digital forensics needs. When you ingest higher-volume, lower-value data into Splunk, it counts against your ingest license meter and drives up your cost of Splunk software ownership.

To alleviate this problem, you might try to drop data using transforms and conf files, but this can be complicated. Dropping data in Splunk Enterprise involves:

  • Memorizing syntax
  • Handwriting stanzas
  • Expensive iteration
  • Editing of many conf files
  • Manual deployment

This is time-consuming and error-prone. With Splunk Cloud Platform, it can be even more complicated and you might need to deploy a custom app to accomplish this.

Therefore, you want a better method to tier your data, sequester lower-value data, and selectively allocate data into cold storage. Doing so still means it's available for compliance and auditing when needed, but it can cost you significantly less money.

Solution

The ingest actions feature in 9.0 provides a new user interface for seamless authoring and deployment of rules for filtering, masking, and routing. A native-built and free feature, ingest actions is available to almost all Splunk Enterprise and Splunk Cloud Platform customers. 

The ingest actions feature is not currently available in GCP Splunk Cloud Platform stacks.

Admins using the Splunk platform 9.0 can find ingest actions under Settings in the global navigation, and then in the Data subsection. You must have the list_ingest_ruleset: list existing rulesets and edit_ingest_ruleset: create / edit rulesets capabilities to create rule sets using ingest actions. Admin and sc_admin roles are automatically granted these capabilities. Ingest actions take place on ingest transformation code. In heavy forwarders and indexers, transforms are applied before rulesets.

Using ingest actions provides the following benefits:

  • A UI to preview and validate rules and logic. The UI reduces iteration time between authoring and deployment in production.
  • Visibility of the conf file's backend if you still want to configure rulesets manually.
  • The ability to drop or filter noisy events like DEBUG logs, so that they do not count against your ingest license meter.
  • The ability to mask fields like PCI and IP addresses for daily search and reporting use cases.
  • The option to route events to Splunk, AWS S3, or both. By routing to S3, you reduce events coming into Splunk software and send them to a cheaper location.

Next steps

Now that you understand the benefits of using the ingest actions feature, watch the demo in this .Conf22 Talk (Introducing ingest actions: filter, mask, route, repeat) and read the related Splunk Docs article Use ingest actions to improve the data input process (Splunk Enterprise or Splunk Cloud Platform) to learn how to implement this feature.

These additional Splunk resources might help you understand and implement this product tip: