Migrating from Federated Search 1.0 to 2.0 for S3 and Security Lake deployments

AWS S3: Customer-Managed

AWS S3: Splunk-Managed Glue Tables

Amazon Security Lake

FedRAMP / IRAP

► Federated Search to S3 (Customer-Managed)

Infrastructure changes

• The unified dataset experience in the Data Management app is activated.
• Legacy Federation to S3 in Admin Settings is available, but creation of new legacy providers and indexes is blocked.
• All new data objects must be created as connections and datasets in the Data Management app.

Language changes

• sdselect remains functional but triggers a warning message:
• SPL2 is enabled and can search over both legacy federated indexes and new federated datasets. In case of a name conflict, SPL2 defaults to the dataset.

Recommended actions

• Explore the Data Management app and create sample connections and datasets for federated search.
• Manually recreate federated providers as connections and federated indexes as datasets. See the Federated Search for Amazon S3 2.0 documentation. The sdselect command can only search over federated indexes, but SPL2 can search over legacy federated indexes and federated datasets. SPL2 defaults to a dataset if it has the same name as the index.
• Manually recreate pipelines and destinations as connections and datasets for Splunk Edge Processor and Splunk Ingest Processor. See Send data from Edge Processors to Amazon S3 and Send data from Ingest Processor to Amazon S3.
• Try the SPL to SPL2 conversion tool to identify and convert saved searches and dashboards using sdselect.

► Federated Search to S3 (Splunk-Managed Glue Tables - SMGT)

Infrastructure

• The unified dataset experience in the Data Management app is activated.
• Legacy Federation to S3 in Admin Settings is available, but creation of new legacy providers and indexes is blocked.
• All new data objects must be created as connections or datasets in the Data Management app.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 does not work with legacy SMGT indexes and returns the error: "Error in 'SearchParser': The SPL2 query is invalid: 'SPL2 is not available for dataset with kind 'Amazon S3 with Splunk managed Glue Table'. Use the SPL 'sdselect' command instead."

Recommended actions

• Explore the Data Management app and create sample connections and datasets.
• Because SMGTs are not supported in FS2.0, manually recreate the index in Glue as a customer-managed data table with a SerDe library. See Creating partitioned AWS Glue tables for CloudTrail datasets.
• The sdselect command can only search over federated indexes, and SPL2 is not available for SMGT indexes.

► Federated Search to Amazon Security Lake (FS-ASL)

Infrastructure

• Legacy Federation in Admin Settings remains available and functions without constraints.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 does not work with ASL indexes and returns the error: "Error in 'SearchParser': The SPL2 query is invalid: 'SPL2 is not available for dataset with kind 'Amazon Security Lake'. Use the SPL 'sdselect' command instead."

Recommended actions

• None.

► Federated Search to S3 (FedRAMP and IRAP customers)

Infrastructure

• Legacy Federation remains available in Admin Settings, but creation of new legacy providers and indexes is blocked.
• The unified dataset experience is not available yet.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 is supported on legacy federated indexes.

Recommended actions

• Use the SPL to SPL2 conversion tool to identify and convert saved searches and dashboards using sdselect.

► Federated Search to S3 (Customer-Managed)

Infrastructure

• Legacy federated indexes become read-only; editing is blocked.
• Legacy federated providers and indexes are automatically migrated to the Data Management app as connections and datasets. Review drafts, configure the new authentication method, and activate the migrated objects.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 is enabled and can search over both legacy federated indexes and new federated datasets, defaulting to the dataset on a name conflict.

Recommended actions

• Review and activate automatically migrated connections and datasets in the Data Management app: set up the new authentication method, review dataset names for conflicts, and activate migrated datasets.
• Manually recreate any legacy pipelines and destinations for Splunk Edge Processor and Splunk Ingest Processor using the Edge Processor and Ingest Processor guides.
• Convert all federated saved searches, scheduled searches, and dashboards from sdselect to SPL2 with the SPL to SPL2 conversion tool. This is the last release where sdselect is supported.

► Federated Search to S3 (Splunk-Managed - SMGT)

Infrastructure

• Legacy federated indexes become read-only; editing is blocked.
• Legacy SMGT providers and indexes are not migrated automatically due to technical constraints with CloudTrail and VPC logs. Recreate these Glue tables manually (see Recommended actions in this section).

Language

• sdselect remains functional but triggers a warning message.
• SPL2 is enabled and can search over both legacy federated indexes and new federated datasets, defaulting to the dataset on a name conflict.

Recommended actions

• Because FS2.0 no longer supports SMGTs, manually recreate the index in Glue as a customer-managed data table with a SerDe library. See Creating partitioned AWS Glue tables for CloudTrail datasets. The sdselect command can only search over federated indexes, but SPL2 can search over indexes and datasets, and defaults to a dataset if it has the same name as the index.
• Manually recreate any legacy pipelines and destinations for Splunk Edge Processor and Splunk Ingest Processor using the Edge Processor and Ingest Processor guides.
• Convert all federated saved searches and dashboards from sdselect to SPL2 with the SPL to SPL2 conversion tool. This is the last release where sdselect is supported.

► Federated Search to Amazon Security Lake (FS-ASL)

Infrastructure

• Legacy Federation remains available and functions without constraints.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 is enabled and can search over both legacy federated indexes and new federated datasets, defaulting to the dataset on a name conflict.

Recommended actions

• Use the SPL to SPL2 conversion tool to convert all federated saved searches, scheduled searches, and dashboards from sdselect to SPL2. This is the last release where sdselect is supported.

► Federated Search to S3 (FedRAMP and IRAP customers)

Infrastructure

• Legacy Federation remains available, but creation and editing of legacy providers and indexes is blocked.
• The unified dataset experience is not available yet.

Language

• sdselect remains functional but triggers a warning message:
• SPL2 is enabled and can search over both legacy federated indexes and new federated datasets, defaulting to the dataset on a name conflict.

Recommended actions

• Use the 
• SPL to SPL2 conversion tool to convert all federated saved searches and dashboards from sdselect to SPL2. This is the last release where sdselect is supported.
• To avoid the gap between legacy FSS3 deprecation and FS2.0 support, consider requesting an exemption from the version upgrade to 10.6.

► Federated Search to S3 (Customer-Managed and SMGT)

Infrastructure

• Federated Search to S3 is removed from the Federation section in Admin Settings.
• Legacy federated indexes are removed from the front-end and federated.conf.

Language

• sdselect is fully removed; search commands will not work.
• SPL2 is the sole search language for external federation.

Recommended actions

• None. All steps should have been taken during 10.4 and 10.5 migration.

► Federated Search to Amazon Security Lake (FS-ASL)

Infrastructure

• Legacy federated indexes remain available until otherwise noted.

Language

• sdselect is removed.
• SPL2 is the sole search language for external federation.

Recommended actions

• None. All steps should have been taken during 10.4 and 10.5 migration.

► Federated Search to S3 (FedRAMP and IRAP customers)

Infrastructure

• Federated Search to S3 is removed from the Federation section; legacy federated indexes are removed from the front-end and federated.conf.
• FS2.0 availability will be announced.

Language

• sdselect is removed.
• SPL2 is the sole search language for external federation.

Recommended actions

• To avoid the gap between legacy FSS3 deprecation and FS2.0 support, consider requesting an exemption from the version upgrade to 10.6.

Step 1: Auditing and taking inventory

• Identify all active legacy federated providers and indexes in the Settings > Federation section.
• Export a list of all saved and scheduled searches currently using the sdselect command.

Step 2: Migrating configuration

• For customer-managed S3: Use the Data Management app to recreate legacy indexes as new connections and datasets. See the Federated Search for Amazon S3 2.0 documentation.
• For SMGT: Follow the manual migration guidance from the product team to recreate indexes in Glue with a SerDe library as customer-managed data tables. See Creating partitioned AWS Glue tables for CloudTrail datasets.
• For routing to S3: Use the Data Management app to recreate pipelines and destinations using the Edge Processor and Ingest Processor guides.

Step 3: Migrating language

• Rewrite: Convert sdselect syntax to SPL2 using the SPL to SPL2 conversion tool.
• Validate: Because SPL2 is backwards compatible with legacy indexes during the 10.4–10.5 transition, test new SPL2 queries against existing legacy indexes to ensure parity before the 10.6 cutoff. Remember, SPL2 always defaults to a dataset in case of a naming conflict with a federated index.

Step 4: Performing final cleanup

• Update all saved searches, dashboards, and alerts to use the new SPL2 syntax and point to the new unified datasets.

Creating an AWS Glue table for a CloudTrail dataset

Run a CREATE EXTERNAL TABLE DDL statement to create your AWS Glue table.

1. Log in to your AWS account and navigate to the Amazon Athena console.
2. Select Query editor from the left-hand navigation pane.
3. In the Data panel, select the AWS Glue Data source, Catalog, and Database for the query. The AWS Glue table you create will be contained by the AWS Glue Database you select.
4. The following CREATE EXTERNAL TABLE DDL statement creates an AWS Glue table for a CloudTrail log dataset. Copy this statement and paste it into the query editor.

   CREATE EXTERNAL TABLE <your_AWS_Glue_table_name> (
       eventversion STRING,
       useridentity STRUCT<
           type:STRING,
           principalid:STRING,
           arn:STRING,
           accountid:STRING,
           invokedby:STRING,
           accesskeyid:STRING,
           username:STRING,
           onbehalfof: STRUCT<
               userid: STRING,
               identitystorearn: STRING>,
           sessioncontext:STRUCT<
               attributes:STRUCT<
                   mfaauthenticated:STRING,
                   creationdate:STRING>,
               sessionissuer:STRUCT<
                   type:STRING,
                   principalid:STRING,
                   arn:STRING,
                   accountid:STRING,
                   username:STRING>,
               ec2roledelivery:string,
               webidfederationdata: STRUCT<
                   federatedprovider: STRING,
                   attributes: map<string,string>>
           >
       >,
       eventtime STRING,
       eventsource STRING,
       eventname STRING,
       awsregion STRING,
       sourceipaddress STRING,
       useragent STRING,
       errorcode STRING,
       errormessage STRING,
       requestparameters STRING,
       responseelements STRING,
       additionaleventdata STRING,
       requestid STRING,
       eventid STRING,
       resources ARRAY<STRUCT<
           arn:STRING,
           accountid:STRING,
           type:STRING>>,
       eventtype STRING,
       apiversion STRING,
       readonly STRING,
       recipientaccountid STRING,
       serviceeventdetails STRING,
       sharedeventid STRING,
       vpcendpointid STRING,
       vpcendpointaccountid STRING,
       eventcategory STRING,
       addendum STRUCT<
           reason:STRING,
           updatedfields:STRING,
           originalrequestid:STRING,
           originaleventid:STRING>,
       sessioncredentialfromconsole STRING,
       edgedevicedetails STRING,
       tlsdetails STRUCT<
           tlsversion:STRING,
           ciphersuite:STRING,
           clientprovidedhostheader:STRING>
   )
   PARTITIONED BY (region string, year string, month string, day string) -- Optional. Set to the partition level your query requires.
   ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
   STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
   OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
   LOCATION 's3://<path_to_your_CloudTrail_data>'; -- A location string that follows this syntax: s3://<bucket-name>/<additional-prefix-folders>/AWSLogs/<AWS-account-ID>/CloudTrail

5. Use the following table to specify various values in the statement you just pasted into the query editor.

   Parameter	Value	Description
   CREATE EXTERNAL TABLE	<your_AWS_Glue_Table_Name>	Required. The name of the AWS Glue table that you want to create with this query.
   PARTITIONED BY	(region string, year string, month string, day string)	Optional. If you want your AWS Glue table to take data partitions into account, provide the partition levels you want to include, separated by commas. For example, if you only want to partition by region and year, put (region string, year string) as the value for PARTITIONED BY. If you define partition levels here, you must also specify the actual partitions you want the query to use when you follow the steps in Adding partition columns to your new AWS Glue table.
   LOCATION	<path_to_your_CloudTrail_data>	Required. The Amazon S3 bucket location for the CloudTrail dataset that the AWS Glue table you're creating will represent. The location file path must follow this syntax: s3://<general_purpose_bucket_name>/<additional_prefix_folders>/AWSLogs/<your_AWS_account_ID>/CloudTrail. The <additional_prefix_folders> might not be present in the location path. These can be one or more additional folders that people optionally set up to differentiate between multiple datasets that are being stored in the same Amazon S3 general purpose bucket.

6. (Optional) Remove any fields not required for your AWS Glue table. If you need to read only a certain set of columns, your table definition can exclude the other columns.
7. When you have finished making changes to the CREATE EXTERNAL TABLE statement, select Run to run the statement and create an AWS Glue table that represents the AWS CloudTrail log dataset.

Adding partition columns to your new AWS Glue table

After you create an AWS Glue table for your CloudTrail dataset, use the ALTER TABLE ADD PARTITION DDL statement to update the table with columns for each partition you want to search. Partitions that are not included will not be searchable when you run federated searches.

1. Log in to your AWS account and navigate to the Amazon Athena console.
2. Select Query editor from the left-hand navigation pane.
3. In the Data panel, select the AWS Glue Data source, Catalog, and Database for the query. The AWS Glue table you create will be contained by the AWS Glue Database you select.
4. The following ALTER TABLE ADD PARTITION statement loads the partitions you define into your AWS Glue table. Copy this statement and paste it into the query editor.

   ALTER TABLE <your_AWS_Glue_table_name> ADD
       PARTITION (region='<region>', year='<year>', month='<month>', day='<day>') -- Set the exact partitions you want the query to search.
       LOCATION 's3://<path_to_your_CloudTrail_data_with_partitions>'; -- An S3 location that follows this syntax: s3://<bucket-name>/<additional-prefix-folders>/AWSLogs/<your-AWS-account-ID>/CloudTrail/<AWS-region>/<year>/<month>/<day>/

5. Use the following table to specify values in the statement.

   Parameter	Value	Description
   TABLE	<your_AWS_Glue_Table_Name>	The name of the AWS Glue table that you want to create with this query.
   PARTITION	(region='<region>', year='<year>', month='<month>', day='<day>')	Specify the actual partitions you need to search to create the table. Must match the partition level defined by the PARTITIONED BY parameter. For example, if you set PARTITIONED BY to (region string, year string), you must set PARTITION to (region='us-west-1', year='2025').
   LOCATION	<path_to_your_CloudTrail_data_with_partitions>	The Amazon S3 bucket location for the CloudTrail dataset that the AWS Glue table you're creating will represent. The location file path must include the partitions that you want to query for your AWS Glue table. The syntax is as follows: s3://<general_purpose_bucket_name>/<additional_prefix_folders>/AWSLogs/<your-AWS-account-ID>/CloudTrail/<AWS-region>/<year>/<month>/<day>/. For example, if you set PARTITION to (region='us-west-1', year='2025', month='04'), you must set LOCATION to s3://my-bucket/AWSLogs/<your-AWS-account-ID>/CloudTrail/us-west-1/2025/04/.

   If you want to load columns for multiple partitions, provide multiple PARTITION parameters with corresponding LOCATION clauses, like this:

   ALTER TABLE mydemotable ADD
       PARTITION (region='us-west-1', year='2025', month='04')
       LOCATION 's3://my-bucket/AWSLogs/<your-AWS-account-ID>/CloudTrail/us-west-1/2025/04/'
       PARTITION (region='us-west-1', year='2025', month='05')
       LOCATION 's3://my-bucket/AWSLogs/<your-AWS-account-ID>/CloudTrail/us-west-1/2025/05/';

6. When you have finished making changes to the statement, select Run to run the query and add columns to your AWS Glue table for the partitions in the dataset that the Glue table represents.

For more information about this method of AWS Glue table creation, see Create a table for CloudTrail logs in Athena using manual partitioning in the Amazon Athena User Guide. For more information about the ALTER TABLE ADD PARTITION DDL statement, see ALTER TABLE ADD PARTITION in the Amazon Athena User Guide.