Diagnosing and fixing issues across the data ingest pipeline

Data source issues

Start here before looking at anything inside the Splunk platform. Ingestion problems are frequently caused by changes at the source that aren't visible in Splunk platform logs until significant delay has accumulated.

Check the following at the data source:

• Endpoint and port. Confirm the application is still sending data to the correct endpoint and port that the Splunk platform listens on.
• Upstream transformations. If a function or process transforms, filters, or routes events before they reach the Splunk platform, for example, a Lambda function in an AWS pipeline, verify that it's still running and hasn't changed its output format.
• Timestamp format. The Splunk platform uses timestamps to place events in time. If the source has changed its timestamp format (for example, from MM/DD/YYYY to DD/MM/YYYY), events will be indexed at the wrong time and appear in unexpected time ranges during search.
• File permissions. If the Splunk platform is monitoring a file or directory, confirm that the forwarder process has read access. Permission changes during OS updates or deployments are a common cause of silent collection failures.
• API rate limits. If the Splunk platform is pulling data from an API, rate limits on the source API might be throttling or rejecting requests.

Forwarding issues

After the data source, check the forwarding tier. Most forwarding issues involve connectivity between components.

• Firewall rules. Confirm that firewall rules allow TCP connections between each forwarder and the next component in the pipeline (whether an intermediate heavy forwarder or the indexer). Firewall changes are frequently the cause of sudden, unexplained ingestion stops.
• Certificate issues. The Splunk platform uses TLS for communication between components. An invalid or expired certificate, or a network device that intercepts the connection and presents an untrusted certificate, will cause the connection to fail silently. The Splunk platform logs the connection failure but doesn't identify the certificate as the cause, so you might need to work with your network team to diagnose and resolve certificate trust issues.
• Credential packages (Splunk Cloud Platform only). Universal forwarders that connect to Splunk Cloud Platform indexers require a credential package to authenticate. These packages expire periodically.
• UDP versus TCP. The Splunk platform can receive data over both UDP and TCP. Because of UDP's connectionless nature, events can be lost before the Splunk platform knows they exist. Where possible, use TCP for more reliable data transfer.

When a forwarding connection fails, you'll see one of these messages in splunkd.log on the forwarder:

• WARN AutoLoadBalancedConnectionStrategy [TcpOutEloop] - Cooked connection to ip=10.0.0.1:9997 timed out — the forwarder can't reach the next component at the specified IP address.
• WARN TcpOutputFd [TcpOutEloop] - Connect to 10.0.0.1:9997 failed. Connection refused — the forwarder can reach the next component, but that component isn't accepting connections. This might mean it's overwhelmed processing existing data or its input queue is full.

Parsing issues

Understanding where parsing happens in your pipeline is important for diagnosing parsing failures. Parsing occurs at the first Splunk platform instance the data passes through:

• If you're using a universal forwarder sending directly to indexers, parsing happens at the indexer.
• If a heavy forwarder is anywhere in the pipeline before the indexer, parsing happens at that heavy forwarder.

Common parsing issues include:

• Timestamp extraction failure. If the Splunk platform can't identify a timestamp in an event, it assigns the time the data was processed, either at the indexer or at the heavy forwarder. Searches scoped to an earlier time range will miss these events entirely.
• Incorrect event boundaries. If the Splunk platform can't determine where one event ends and the next begins, it might merge multiple log lines into a single event or split one event across multiple. Both distort search results.
• Event truncation. If an event exceeds the truncation limit configured for its source type, you'll see only a portion of the event in search results.
• Data routing. The Splunk platform can route events to different indexes or source types using props.conf and transforms.conf. Misconfigured routing rules can silently drop events or send them to an unexpected index, making them invisible in the index you're searching.

Indexing issues

If data has reached the indexer but isn't appearing in searches as expected, check the following:

• Resource constraints. An indexer processing an unexpected burst of data, or running expensive searches at the same time as heavy ingestion, might delay indexing. Events will eventually appear, but not in real time.
• Searchable retention. Each index has a searchable retention period. If you're searching a time range that falls outside that period, results will be empty even if the data was indexed. Confirm your search time range is within the index's retention window.
• Knowledge object permissions. Knowledge objects such as lookups, event types, and field extractions can filter or transform events at search time. If a user doesn't have access to a knowledge object that the search depends on, those events won't appear in their results, even though the underlying data is indexed.

Monitoring queue depth with metrics logs

The Splunk platform writes queue statistics to metrics.log at regular intervals. Each queue entry includes:

• group=queue — identifies the record as a queue metric
• name — the queue name (parsingQueue, aggQueue, indexQueue, tcpout_genericqueue, and so on)
• current_size — how much data is currently in the queue
• max_size — the queue's configured capacity
• blocked — whether the queue is currently blocked from sending to the next stage

You can run a search against metrics.log to chart queue behavior over time:

index=_internal source=*metrics.log sourcetype=splunkd host=<hostname> (name=aggqueue OR name=indexqueue OR name=parsingqueue OR name=splunktcpin OR name=typingqueue OR name=*tcpout*)
| eval ingest_pipe=if(isnotnull(ingest_pipe),ingest_pipe,"none")
| search ingest_pipe=*
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size), curr=if(isnotnull(current_size_kb),current_size_kb,current_size),
    fill_perc=round(((curr / max) * 100),2)
| timechart minspan=30s p90(fill_perc) by name useother=false limit=15

The charts in the following sections below show the four queue saturation states you might see when running this search. Select each state to see what it looks like and what to do.

► Low saturation — healthy baseline

Queue current_size stays well below max_size. You might see blocked=true occasionally, but the queue recovers quickly. This is normal behavior under any load.

There's no backlog and no ingestion delay. The instance has spare capacity to absorb spikes in data volume.

► Busy but healthy

Queue current_size is closer to max_size, and you might see more frequent blocked=true entries. As long as data is still flowing through without delay, this is not a problem.

Monitor this state if you're planning to increase data volume. If the forwarder is already running near capacity, an increase in volume might push it into saturation.

► Saturated with throughput

Queues are at or near 100% capacity. blocked=true is constant. You're seeing ingestion delay and data is not arriving in real time. However, data is still flowing out, and CPU and memory on the instance are not maxed out.

This pattern often means the instance is being throttled. Universal forwarders have a default outbound throughput limit of 256 KBps (maxKBps = 256). If the forwarder is consistently sending at or near this rate, you'll see this saturation pattern even though the instance has available capacity. Check metrics.log for throughput records to confirm.

The throughput limit is configured in limits.conf under the [thruput] stanza, not in outputs.conf. To increase the limit, create or update limits.conf with [thruput] / maxKBps = <value>. Heavy forwarders and indexers default to unlimited (maxKBps = 0). For more information, see the troubleshooting information in Event indexing delay.

When the limit is hit, you'll see this in splunkd.log:

Current data throughput (### kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf

When the saturated queue is blocking data from reaching the parsing queue, you might also see:

WARN TailReader [tailreader0] - Could not send data to output queue (parsingQueue), retrying...

If increasing the throughput limit doesn't resolve the saturation, you can also increase the queue's maxSize. Be aware that larger queues consume more memory, so confirm that the instance has the available memory before making this change.

You can also increase the number of parallel ingestion pipelines to process more data concurrently. This increases CPU usage, so follow the guidelines for your deployment before raising this value.

For more guidance on these settings, see Performance tuning the forwarding tier.

► Saturated without throughput

Queues are at or near 100% capacity, throughput has collapsed to zero, and CPU and memory on the instance are maxed out. Data has stopped flowing entirely.

This state indicates the instance doesn't have enough capacity to handle the current data volume. Consider scaling the instance vertically (more CPU and memory) or adding additional forwarders to distribute the load.