Improving your data management with SPL2 pipelines

Whether you're a Splunk Enterprise or Splunk Cloud Platform customer, the Splunk data management experience will improve the way you onboard, transform, and archive data, helping you get more value from the Splunk platform. Data management is accomplished through a single, modern UI experience (on a search head for on-premises customers or in your cloud stack) that includes an SPL2 Pipeline editor with more than 50 data source templates for a quick start, deployment through Splunk Edge Processor or Splunk Ingest Processor, and monitoring and troubleshooting assistance.

This article focuses on the basics of the SPL2 pipeline, but before we begin, let's do a quick review of the management deployment options. Which one you use impacts your pipeline creation. 

Splunk Ingest Processor

Available to Splunk Cloud Platform customers, Ingest Processor applies filtering, masking, transforming, and converting functions at the time of data ingestion. Then it routes your data to Splunk Cloud Platform, Splunk Observability Cloud, S3 buckets, or Azure, according to your pipeline.

Splunk Edge Processor

Available to both Splunk Cloud Platform and Splunk Enterprise customers, Edge Processor applies filtering, masking, transforming, and converting functions, before routing data to the Splunk platform, S3 buckets, or Azure, according to your pipeline. It is customer-hosts, which gives you control at the edge before data leaves your network, a benefit for security-conscious environments. 

function clean_up_empty_events($source: dataset): dataset {
   return  | from $source
   | eval array_null_keys=filter(json_keys(_raw), json_extract(_raw, $it) == "")
   | eval _raw = json_delete(_raw,array_null_keys)
   | fields - array_null_keys
}
$pipeline = | from $source | eval index = "json_filtered"
| clean_up_empty_events
| into $destination;