Skip to main content
Lantern Home

 

Splunk Lantern

Evaluating the benefits of an intermediate forwarding tier

  1. Last updated
  2. Save as PDF

When you first start working with the Splunk platform, the standard guidance can seem straightforward: send data directly from forwarders to the indexing tier and avoid an intermediate forwarding layer whenever possible.

Blog posts such as Universal or heavy, that is the question? describe an intermediate forwarding tier as an artificial bottleneck that increases the time from event generation to availability for searching, and can also cause data imbalance on the indexing tier that reduces search performance.

However, asynchronous forwarding was added in Splunk platform version 7.3.6, and when properly tuned, it significantly improves data balance (as mentioned in Performance tuning the forwarding tier and the presentation Forwarder data balance). According to the intermediate forwarding documentation, asynchronous load balancing is an officially supported approach that enhances both indexer ingestion performance and search performance.

This raises a question: what happens if you don't use an intermediate heavy forwarder tier and instead send data directly from the universal forwarders to the indexers? This article shows you the results of testing this approach in both cloud-based and on-premises environments, and provides criteria for deciding when an intermediate forwarding tier is appropriate for your deployment.

The findings in this article reflect specific testing carried out by the author, Gareth Anderson, in his own environments, so your results might vary depending on the number of forwarders, network topology, data volumes, and tuning of outputs.conf. You should treat these results as a reference point and benchmark against your own environment before making architectural decisions.

Testing the idea

The existing on-premises environment included an intermediate heavy forwarder layer. For a new cloud-based environment, which had only a limited number of forwarders, this tier was eliminated and both heavy forwarders (HFs) and universal forwarders (UFs) were configured to forward data directly to the indexing tier through asynchronous forwarding.

The balance per-indexer was then measured on a few select indexes:

| tstats count where index=<index> host=* groupby splunk_server
| eventstats sum(count) AS total
| eval perc=round((count/total)*100,2)

In the new cloud-based environment of 8 indexers, the percentage of data per-indexer was as high as 23.28%, and as low as 4.12%. Clearly, this environment requires additional tuning, but changing the outputs.conf settings was complicated by change‑control processes and the limited direct control over downstream forwarders.

On‑premises, an intermediate HF tier was still in place. The indexer cluster in question has 20 indexers, and using the query above, roughly 4.96% to 5.05% of total data was observed landing on each indexer — a substantially better balance than the cloud-based environment without an intermediate tier.

When is the intermediate tier appropriate?

Returning to the original question, should you have an intermediate forwarding tier? Use the following criteria to help you decide:

  • Do you have so many forwarders that asynchronous forwarding becomes impractical? With thousands of forwarders maintaining numerous open connections to the indexing tier, performance issues are likely. This has been tested with more than 100 HFs, but not beyond that scale.
  • Do you have the ability to tune outputs.conf on downstream forwarders? If not, changes to asynchronous forwarding settings will be more difficult to apply and will generally complicate tuning.
  • Are you comfortable applying tuning changes across a large number of forwarders? An intermediate tier reduces the number of places where configuration must be managed.

If the answer to any of the above criteria is yes, then an intermediate forwarding tier is likely to be of benefit.

As noted in the community discussion Wrongly merged Events/permanently blocked tcpout queue with intermediate universal forwarder, it might be safer to use an intermediate HF tier. If you do use a universal forwarder, you might achieve a similar result using EVENT_BREAKER, although this has not been tested.

Conclusion

If data balance across the indexing tier is a priority, an intermediate forwarding tier will likely help.

However, if you have full control over all forwarders and the total number is small enough that asynchronous forwarding remains practical, the intermediate tier might offer limited benefit beyond reducing the number of locations where outputs.conf must be tuned.

Additional resources

These resources might help you understand and implement this guidance: