Skip to main content
 
 
 
Splunk Lantern

Configuring file system destinations with ingest actions

 

Customers with a 9.3.0+ version of Splunk Enterprise can access file system destinations for ingest actions. This will give you more control over where data lives and enable Splunk admins to flexibly optimize their storage options for different data tiers.

How do I access and use file system destinations?

Navigate to Settings > Ingest Actions. In the Destinations tab, there is a “File System” option when creating a new destination. The Splunk platform supports up to 8 file system destinations on top of the already supported 8 Amazon S3 locations, for a total of 16 external output destinations.

unnamed - 2024-08-06T103711.874.png

In the GUI wizard, create a Destination Title and Directory Path for writing events to and adjust any of the default settings as necessary.

For more information on any of the destination settings, check out the RFS Output stanza in the outputs.conf spec.

unnamed - 2024-08-06T103750.752.png

Files are written to by continuously appending the output file until it reaches the default specified in the appendToFileUntilSizeMB setting. This setting is adjustable in the destination configuration UI, in addition to the backend .conf file directly. Files are partitioned by day, and in each day folder, events are separated by hour.

unnamed - 2024-08-06T103825.331.png

Because the events are formatted in the same way as an ingest actions Amazon S3 output (new line delimited JSON with default fields emitted at minimum), you can selectively upload data (without any additional ETL steps) to an accessible Amazon S3 location and access it using Splunk federated search for S3.

unnamed - 2024-08-06T103905.679.png

What if I want to mount a file system with Network File System (NFS)?

If you want to mount a file system using NFS to enable distributed file sharing, use any NFS v4.1 or higher, provisioned with read/write access.

Step 1: Set up the NFS server

  1. Switch to the root account to make system changes.

    sudo su

  2. Install the necessary packages for the NFS server.

    sudo apt update

    sudo apt install nfs-kernel-server

  3. Create an output directory.

    sudo mkdir /lorem/ipsum/nfs_server -p

  4. Change directory ownership to allow NFS to read/write to the folder of your choice.

    sudo chown nobody:nogroup /lorem/ipsum/nfs_server

  5. Open the exports file and enable the client IP to access the server directory of interest.

    sudo vi /etc/exports

    /lorem/ipsum/nfs_server

    10.224.24.5(rw,sync,no_subtree_check,no_root_squash,all_squash,insecure)

  6. Restart the NFS server to update settings.

    sudo systemctl restart nfs-kernel-server

  7. Check the firewall status to know if it is enabled and what is allowed. If disabled, there is no need to do step 8. If the firewall is enabled and NFS traffic is not allowed (traffic from the client to server), then allow it by following step 8.

    sudo apt install ufw

    sudo ufw status

  8. Open port 2049 at the server to allow traffic from the client IP.

    sudo ufw allow from 10.224.24.5 to any port nfs

Step 2: Set up the NFS client

  1. Switch to the root account to make system changes.

    sudo su

  2. Install the necessary packages for the NFS client.

    sudo apt update

    sudo apt install nfs-common

  3. Create the NFS client directory.

    sudo mkdir -p /lorem/ipsum/fs_local

  4. Mount the client directory with the server directory.

    sudo mount 10.224.58.143:/lorem/ipsum/nfs_server /lorem/ipsum/fs_local

Finally, set up a new file system destination in ingest actions with the nfs-client path (for example, /lorem/ipsum/fs_local). Any rulesets that write to the local client should now feed into the NFS server directory as well.

unnamed - 2024-08-06T104652.086.png

Next steps

This feature further enables data optimization and data tiering use cases for on-premises and hybrid customers by expanding choice in output destinations. We hope you take advantage of this new capability and share with us any feedback or feature requests.

These additional resources might help you understand ingest actions and implement data reduction strategies: