Many of the Splunk Add-on for Unix and Linux data collection inputs are scripted inputs. The scripts smooth over variations to the input that arise across various Unix and Linux platforms. Scripted input means that Splunk runs the same commands you run on the terminal.

If a needed Unix command is not installed or lacks proper permissions, then Splunk can't collect the associated data. In other words, the Unix commands and access to them are prerequisite for collecting data. See the scripted inputs table in the Splunk Add-on for Unix and Linux manual for the underlying UNIX commands.

There are a few inputs in the Splunk Add-on for Unix and Linux that monitor files. You can guidance on setting these permissions in Setting unix permissions for file monitoring.

Troubleshooting

As you deploy the Splunk Add-on for Unix and Linux across many hosts, you may find it impractical to check each and every server for errors as the scripted inputs run. Luckily, Splunk excels at solving that very problem!

Apply the Best practice: Forward search head data to the indexer layer principle not just to the search heads, but to any add-on deployment destination. If you're worried about the amount of disk space this might consume, then follow this practice only during the process of deploying the Splunk Add-on for Unix and Linux.

Once those instances are forwarding their Splunk data, it should be trivial to find issues within the _internal index. Try this search to identify problem hosts:

index=_internal sourcetype=splunkd source=*splunkd.log log_level=ERROR component=ExecProcessor Splunk_TA_nix
As you pinpoint issues, use the instructions outlined in Troubleshoot the Splunk Add-on for Unix and Linux to debug further.

Warnings for Windows deployment servers

If you're using a Windows system as a deployment server, you may experience end-of-line (eol) issues once the scripted inputs of Splunk_TA_nix attempt to execute on the UNIX clients. Each line in a text file on Windows ends with a carriage return ( \r) and then a new line ( \n), whereas on UNIX, lines end only with the new line ( \n).

Splunk is able to reconcile the differences when it reads the configuration files. But the UNIX shells themselves will throw all sorts of misleading errors rather than ignoring the carriage returns. A symptom of this might look like the thread described in the Splunk Answers post Splunk_TA_nix cannot open scripts.

Resolve this by converting the line endings to UNIX format at its location on the deployment server Windows host. See the Splunk Answers topic Splunk_TA_nix cannot open scripts for solutions on how to fix the files on the Windows machine before the deployment server distributes to the clients. Keep in mind that manual text edits of those files may reintroduce the problem.

Another good Splunk Answers topic for Windows considerations is What are the pain points with deploying your Splunk architecture on Windows OS

Configure the Deployment Server for scale

Rather than listing out every UNIX host in the deployment server, embrace the wildcards ( *) in the machineTypesFilter. For details, see set up client filters in the Updating Splunk Enterprise Instances manual.

As your distribution becomes more distributed, you may find it useful to review the Splunk Answers topic Deploying identical add-on with slightly different configurations.