Understanding file monitoring on Splunk Forwarders: The basics of fishbucket and CRC

1. File tracking with fishbucket and CRC 

When a Forwarder discovers a monitored file, it records how far it has read that file. 

Fishbucket is the internal checkpoint database that stores this read state. 

For example, when monitoring /var/log/app.log, the Splunk platform must remember how much of the file it has already read. Without this information, the same data could be ingested repeatedly after every Splunk restart or log rotation. 

In a standard configuration, fishbucket is located at: 

$SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db 

If SPLUNK_DB has been moved from its default location, check the actual data storage path. 

The Splunk platform does not identify files by filename alone. 

By default, a Splunk monitor input reads the first 256 bytes of a file and calculates a CRC from that content. This beginning CRC acts like a fingerprint based on the beginning of the file. 

The Splunk platform uses the beginning CRC as a key to look up previously stored information in fishbucket. When it finds a matching entry, it refers to values such as seekAddress, which represents the byte position already read, and seekCRC, which verifies the content at that position. 

CRC therefore helps the monitor input identify a file and retrieve its stored read position from fishbucket. The same information is used when the Splunk platform identifies files during log rotation. This prevents the Splunk platform from treating a rotated file as a new file and ingesting the same data twice. 

However, two different files will have the same beginning CRC if their first 256 bytes are identical. When the beginning CRC matches, the Splunk platform uses the seekAddress and seekCRC from the same fishbucket entry to determine continuity with the file it read previously. 

If different files share the same beginning CRC, the Splunk platform cannot track their read positions independently. As a result, one file might not be read, or its events might be ingested with an unexpected source. 

Pay particular attention to files with characteristics such as: 

• A long, fixed header at the beginning 
• The same stack trace or fixed message at the beginning 
• Multiple log types generated from the same template 

In these cases, the default beginning CRC might not sufficiently distinguish the files. A file can appear to have been skipped, or events can appear under an unexpected source. CRC is an important input to file identification, but the first 256 bytes are not always enough to distinguish every file. 

2. CRC-related logs and Read status

Problems related to fishbucket or CRC can produce characteristic messages and states in splunkd.log and inputstatus. For example, splunkd.log might contain: 

File will not be read, is too small to match seekptr checksum 

Last time we saw this initcrc, filename was different. 

You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. 

This indicates that fishbucket contains a previous entry matching the target file's beginning CRC, but the current file is too short to validate seekCRC at the stored read position. It can occur when a file is truncated or recreated, or when the Splunk platform detects a different file whose beginning is identical. 

inputstatus might also report: 

type = ignored file (crc conflict, needs crcSalt) 

This can indicate that another file has the same beginning CRC and the Splunk platform cannot treat the target as an independently tracked file. 

The following finished reading state indicates that the Forwarder's monitor input has processed the file through its end: 

file position = 4738 
file size = 4738 
percent = 100.00 
type = finished reading 

However, this state alone does not prove that the events were stored on an indexer and are searchable. 

When finished reading is present, determine whether the data became unavailable during forwarding, indexing, or because of the search conditions. 

3. Configuring initCrcLength and crcSalt 

initCrcLength and crcSalt are both inputs.conf settings related to CRC-based file identification. Choose between them based on where the files begin to differ and how the logs are rotated. 

initCrcLength specifies the number of bytes from the beginning of a file that the Splunk platform uses to calculate the CRC. It is useful when the first 256 bytes are identical but differences appear later in the files. 

[monitor:///path/to/logs/*.log] 
initCrcLength = 1024

A larger value is not automatically better. Compare sample files and determine approximately where their content begins to differ before choosing a value. 

Changing initCrcLength also changes the CRC used for file identification, so existing data might be ingested again. Confirm the target files and potential impact before applying the change in production. 

If multiple files share a long fixed header or stack trace, increasing initCrcLength might not have the intended effect. In that case, review crcSalt or the monitor stanza design. 

crcSalt adds a salt value to the CRC calculation. A common configuration is: 

[monitor:///path/to/logs/*.log] 
crcSalt = <SOURCE>

When <SOURCE> is specified, the full file path is added to the CRC. Files with identical beginning content are therefore identified separately when their paths differ. 

This is a possible solution when separate files have identical first 256 bytes. 

However, crcSalt = <SOURCE> is not safe in every situation. If log rotation or file movement changes the path, the Splunk platform can identify the file as a different file and ingest data it has already read. 

Use particular care in situations such as: 

• app.log is renamed to app.log.1. 
• A wildcard stanza monitors both the active and rotated files. 
• Rotated files remain in the same directory. 

If the operational design ensures that the active file is read before rotation, consider excluding rotated files from monitoring with blacklist. If different log types are monitored by separate monitor stanzas and their CRCs conflict across those stanzas, another option is to assign a different fixed crcSalt value to each stanza. 

[monitor:///path/to/system*.log] 
crcSalt = system 
[monitor:///path/to/journal*.log] 
crcSalt = journal 

A fixed salt distinguishes CRCs between stanzas that use different values. It does not distinguish files within the same stanza, because those files receive the same salt. 

Adding crcSalt to an existing monitor stanza, or changing its value, also changes the CRC used for file identification. Test the change to ensure that existing data is not ingested again. 

4. Commands and search examples for troubleshooting 

On the Forwarder, first use the following commands to check the monitoring configuration and read status: 

$SPLUNK_HOME/bin/splunk list inputstatus 
$SPLUNK_HOME/bin/splunk btool inputs list --debug 

On a Heavy Forwarder that performs parsing, check the effective props.conf configuration when necessary: 

$SPLUNK_HOME/bin/splunk btool props list --debug 

In splunkd.log, search for strings such as: 

TailReader 
TailingProcessor 
WatchedFile 
FileClassifierManager 
File will not be read 
seekptr checksum 
initcrc 
crc conflict 
needs crcSalt 
Checksum for seekptr 
File too small to check seekcrc 
Will re-read entire file

When searching indexed data, temporarily remove the source condition and check the log text and _indextime: 

index=<target_index> "<unique raw string>" 
| table _time _indextime host source sourcetype index splunk_server _raw

The following checks can help narrow down the issue: 

• Whether the event remains missing after removing the source condition 
• What the results show when using _indextime instead of _time 
• Whether the event was ingested into a different source, sourcetype, or index 
• Whether event breaking caused the data to be included in the _raw of another event 

If the file is finished reading but the event still cannot be found, expand the investigation beyond the Forwarder's file-reading stage and check forwarding, indexing, and the search conditions in that order.