Splunk Edge Processor is the latest innovation in data preprocessing. It offers more efficient, flexible data transformation – helping you reduce noise, control costs, and gain visibility and control over your data in motion. It works at the edge of your network. You can use it to filter, mask, and transform your data close to its source before routing the processed data to the environment of your choice.
With Edge Processor, you can:
- Filter low-value or noisy data, like DEBUG logs.
- Enrich and extract only the critical data.
- Route different “slices” of data to Splunk platform and Amazon S3.
Benefits and value of Splunk Edge Processor
- Reduce noise
- Reduce costs associated with data ingest
- Enrich data with more context
- Mask sensitive data (for example PII)
- Reduce log volume and size
- Gain increased visibility into streaming data
- Reduce search time
- More efficient, flexible data transformation
- Centralized control through cloud control plane
- Leverage SPL2 for advanced data processing
- A guided pipeline builder to simplify data routing
- Computes at a much faster rate, with fewer compute resources required compared to ingest actions or heavyweight forwarders
Splunk Edge Processor is included with your Splunk Cloud Platform, available at no additional cost. Learn more about the requirements to use Edge Processor and how to request access if you do not already have it.
How Splunk Edge Processor works
Splunk Edge Processor combines Splunk-managed cloud services, on-premises data processing software, and SPL2 to support data processing at the edge of your network. It allows you to ingest data into Splunk, S3 or other systems. This service offering is delivered through the cloud control plane, with an Edge Processor node installed in the customer infrastructure for data processing (i.e. data plane). Learn more about the Edge Processor system architecture.
Using simple-to-deploy nodes, Splunk Edge Processor allows you to filter, route and process data generated by Splunk Forwarders and other sources before it is ingested into Splunk Enterprise or Splunk Cloud Platform. You define where you want to deploy the Edge Processor nodes, as well as the Edge Processor node name, description and tags.
When Splunk Edge Processor nodes are deployed, you control the destination to where your Edge Processors and pipelines send data. You can also configure a “default destination” per Edge Processor node to route unprocessed data. If you don't specify a default destination, Edge Processors will drop unprocessed data by default.
As a user, you can easily configure SPL2 based pipelines to filter, mask, transform, and route data to destinations using the pipeline builder. It supports most SPL2-based commands for pre-ingest data processing (e.g., regex, eval, etc). Learn more about SPL2 profiles and view a command compatibility matrix by product for SPL2 commands and eval functions.
- Supported sources: S2S, HEC, and syslog.
- Supported destinations: S2S, HEC, and AWS S3 currently.
The current statuses of the capabilities and limitations of Edge Processor are:
- Supported actions: Filtering, transforming, masking, routing (stateless, lightweight operations)
- In the roadmap: Lookups, dynamic sampling, dedup, summarizing, and stats functions
- Not supported: Data encryption/decryption
How to get started
Login to Splunk Cloud Platform and navigate to Splunk Data Management console to start using Splunk Edge Processor today. You only need to copy and paste a command line into your Linux machine to install your first Splunk Edge Processor node.
You can access Splunk Edge Processor in the following ways:
- If using Splunk Web UI, from the homepage, click Settings > Add data > Edge Processor.
- You can also directly navigate to the Edge Processor using the following link: https://px.scs.splunk.com/<your Splunk cloud tenant name>/data-management/
Review the additional resources below, then head to the next step to configure and deploy your first Splunk Edge Processor with step-by-step guidance.
- Join the #edge-processor Slack channel for direct support (request access: http://splk.it/slack)
- Blog: Introducing Edge Processor: Next gen data transformation
- Tech Talk: Introducing Edge Processor
- .conf23: Getting Data in more efficiently using the Splunk Edge Processor (session slides)
- Blog: Data preparation made easy: SPL2 for Edge Processor
- Blog: Addition of syslog in Splunk Edge Processor supercharges security operations with Palo Alto firewall log reduction
- Stay up-to-date with Edge Processor Release notes