Using federated search for Amazon S3 (FS-S3) with Edge Processor

Prerequisites

You should ensure you are familiar with Amazon S3, AWS Glue, Amazon Athena, Splunk’s ingest actions, and Splunk’s federated search for Amazon S3. If any of these topics are not familiar, consider taking a few minutes to review them or make sure the documentation is handy. You can also find additional information about partitioning in the article Partitioning data in S3 for the best FS-S3 experience.

Simple pipeline and output

Splunk Edge Processor provides the capability to route data to S3 through defined pipelines. Configuration of EP and authoring pipelines are well-defined in the documentation so we won't repeat them here, but we will show syntax used in the following examples.

In the example pipeline below, we take a data source ($source) and route it to S3 ($destination). Both $source and $destination are defined during the pipeline creation.

pipeline=
| from $source
| into $destination

The Splunk Edge Processor documentation describes how EP writes data to S3 in the following bucket and folder structure:

<bucket_name>/<folder_name>/<year>/<month>/<day>/<instance_ID>/<file_prefix>-<UUID>.json

EP supports the use of an optional prefix prior to the year key that might be useful in reuse of a bucket for multiple subsets of data, applying different partition schemes, and supporting individual Amazon S3 lifecycle rules. File prefixes can be used to describe the data further, but are not available as a partition.

The folder name and file prefix are optional keys. In this example, we’ll omit the folder and prefix resulting in data output in the following format:

<bucket_name>/<year>/<month>/<day>/<instance_ID>/<UUID>.json

Remove implicitly added fields

Splunk Edge Processor might implicitly write fields like path and other metadata information to output to S3. It is recommended to remove implicit fields to maintain control of the schema written to Amazon S3 by EP and to prevent special characters from impacting table creation. Implicit fields can include special characters in the key names such as ':' that need special handling to be supported by the AWS Glue catalog. Specifically, Splunk protocol metadata has the format of key::value.

The example below shows an event that was written to S3 using the using the minimal from $source | into $destination syntax, showing in red an example of an included implicit field:

{
  "time": "1720117251.000",
  "host": "ip-10-0-0-81.ec2.splunkit.io",
  "source": "/var/log/audit/audit.log",
  "sourcetype": "linux_audit",
  "index": "default",
  "fields": { "_path": "/var/log/audit/audit.log" },
  "event": "type=CRYPTO_KEY_USER msg=audit(1720117251.966:268): pid=9306 uid=0 auid=2023 ses=1 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=from-server spid=9322 suid=2023 rport=58640 laddr=10.0.0.81 lport=22  exe=\"/usr/sbin/sshd\" hostname=? addr=10.0.0.59 terminal=? res=success'\u001dUID=\"root\" AUID=\"myuser\" SUID=\"myuser\""
}

The following example contains a pipeline, shown in red, that removes the _path and any implicit fields with key names that include '::':

$pipeline=
| from $source
| where _raw.fieldName=”some value”
| fields - ‘_path’,’*::*’
|into $destination

An example event written to S3 should look like:

{
  "time": "1720121403.000",
  "host": "ip-10-202-39-169.ec2.splunkit.io",
  "source": "/var/log/audit/audit.log",
  "sourcetype": "linux_audit",
  "index": "default",
  "fields": {},
  "event": "type=CRED_DISP msg=audit(1720121403.300:358): pid=29574 uid=0 auid=2023 ses=4 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct=\"root\" exe=\"/usr/bin/sudo\" hostname=? addr=? terminal=/dev/pts/1 res=success'\u001dUID=\"root\" AUID=\"myuser\""
}

Index-time fields

Index-time fields in the JSON payload might be defined as columns by the AWS Glue crawler or DDL, allowing these fields to be searchable in FS-S3 queries. Use eval statements in your pipelines to create new fields in events.

Extracting fields from JSON events can also be accomplished using json_extract and rex commands.

$pipeline=
| from $source
| eval indextimefieldx = json_extract (_raw, “<your_json_field>”),
| fields - ‘_path’,’*::*’
|into $destination

{
“time”:”<epoch time>”,
“host”:”<host>”,
“source”:”<source>”,
“Sourcetype”: “<sourcetype>”,
“index”: “<index>”,
“fields”: {
“<indextimefieldx>”: “<value>”,
},
“event”:”<event string>”
}

Any new fields created using Edge Processor will automatically become indexed extractions at index time. Indexed extractions, especially high cardinality fields, can significantly increase the size of your indexes.

Also keep in mind that data for FS-S3 consumption should be written according to a schema shared by all records written to the Amazon S3 location you will query with FS-S3.

If shared modules are used to build pipelines that operate on the same source type, special care must be taken to remove the extra fields or explicitly build the desired schema prior to sending to the Splunk platform or Amazon S3.

Writing your data to a temporary location to ensure the fields you expect in your events is recommended.

Pipeline updates and deployment considerations

When updating Splunk Edge Processor pipelines used to write data to Amazon S3, use dedicated test pipelines and temporary S3 locations to test and review output. This will help avoid unintended data output in production storage locations and limit interruptions to data collection.

Also, keep in mind that deployment of Edge Processor pipelines occurs in place and deployment is not coordinated when pipelines are deployed across multiple instances. This might result in ingestion interruption while pipelines are deployed. In multiple-instance deployments, deploying new pipelines to new EP instances, then shutting down EP instances that run the older pipelines is one way to help eliminate ingestion interruptions.

Glue catalog considerations

CREATE EXTERNAL TABLE `my_table_name`(
  `time` string, 
  `host` string, 
  `source` string,
  `sourcetype` string,
  `index` string,
  `fields` string,
  `event` string, 
)
PARTITIONED BY ( 
  `year` string, 
  `month` string, 
  `day` string, 
  `instanceid` string)

ALTER TABLE <REPLACE-NAME> ADD PARTITION (year = 'REPLACE', month = 'REPLACE', day=’REPLACE', instanceid='REPLACE');

WITH SERDEPROPERTIES (
  "mapping.my_field_name"="my:field-name"
  )

WITH SERDEPROPERTIES (
  "dots.in.keys"="true"
  )

Splunk platform configuration considerations

Search your data

After you have your federated provider and federated index defined, you can start searching your S3 data using Federated Search for Amazon S3. The example below shows a basic sdselect command returning events from S3.

Service limits

When planning your architecture, you'll need to consider service limits. Review the latest Edge Processor and federated search for S3 documentation for the latest service limits.

Next steps

There are many technical details outside the scope of this article that might influence your decision making when implementing these concepts at scale. Documentation for Federated Search for S3 and Edge Processor (EP) are a good starting point for expanding your knowledge past the concepts discussed in this article. Error handling and data destination availability are particularly important as you consider EP and FS-S3 for your use cases. Familiarize yourself with these behaviors before implementing EP and FS-S3 in a production environment.

If you need additional technical implementation guidance, engage your Splunk and/or AWS account teams.