Migrating from intermediate forwarders to Edge Processor

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

As organizations evolve their data management strategies, a common question arises: Can existing intermediate Splunk Heavy Forwarders be replaced with Splunk Edge Processor?

This guide provides comprehensive context, common scenarios, and key considerations to help you evaluate whether migrating to Edge Processor is right for your environment. It is important to understand that individual environments are frequently nuanced, and there might be several different implementations and requirements across different intermediate deployments within the same organization. Any migration should be carefully considered, and guidance from Splunk On-Demand Services, Splunk Professional Services, or Splunk Architects should be leveraged to properly scope and plan your migration.

Splunk is in the process of mitigating many of the risks and challenges discussed in this article.

Understanding the benefits and considerations

Before embarking on a migration, it's essential to understand both the key motivators and potential challenges associated with moving from intermediate forwarders to Edge Processor.

Key motivators for Edge Processor migration


Motivation	Description
Centralized configuration management	Manage all pipeline configurations from a single, centralized location rather than distributing configurations across multiple forwarders.
Authoring rules in SPL2	Leverage the power and flexibility of SPL2 for creating transformation rules and pipelines.
Improved efficiency	Better scaling of event processing for similar hardware resources, allowing you to do more with your existing infrastructure.
Advanced transformation capabilities	SPL2 provides the ability to run more complex transformations than traditional props/transforms configurations.
UI-based management	Entirely UI-based configuration and management, which is beneficial for people who prefer visual interfaces over configuration file management.

Key considerations before migration


Consideration	Description
Timestamping and parsing relocation	In many cases, timestamping logic will move to the indexing tier, which represents a fundamental change in how events are processed. Additionally, transforms that might otherwise have been ignored on the indexers might begin being processed. Detailed props and transforms review is required. Read more here.
Increasing indexer workload	Moving parsing to the indexers can increase SVC/indexer workload for indexing pipeline and index-time field extractions.
Source type handling	You will need to operate on original source types, which might be different than indexed and search-time source types that your organization is accustomed to working with.
Learning curve	New language (SPL2) and product architecture to learn, which requires investment in training and ramp-up time.
UI-based management	While beneficial for some people, the entirely UI-based approach might be less preferred by administrators who are comfortable with configuration file management.
Differences in data resilience	useAck, persistent queues, and full queue have different behavior on each platform.
Cost	Like Heavy Forwarders, Edge Processors have no licensing cost or usage limitations.

Migration discovery: Evaluating your environment

The following questions and considerations should be used to understand your scenario and help qualify a Splunk Edge Processor migration. The considerations listed are provided to provide context to the questions, not to fully qualify the outcome. Edge Processor migrations from Universal Forwarders and heavy forwarders are nuanced, and the decision of whether to migrate will likely be weighed against the outcomes of these questions as a whole. In some cases, the answers to these questions can rule out a migration entirely, but more often the answers shape the migration.

Understanding your motivation

Question: What is your motivation for implementing Edge Processor or Ingest Processor?

Considerations:

Are you looking to reduce data ingest volumes?
Is easing management overhead a primary driver?
Do you need to transform events in ways that are difficult with current tools?
Are there specific functionalities found in SPL2 that you want to leverage?

Understanding your core motivation will help determine whether Edge Processor is the right solution and will shape the migration approach.

Evaluating current heavy forwarder functionality

Question: How are your heavy forwarders currently configured and used?

Considerations:

Heavy forwarders with or without the use of Ingest Actions can perform a great deal of transformation and routing, including filtering and masking through the use of props and transforms.

Are they primarily used for network funneling?
Are they performing event parsing? Line breaking, and timestamping?
Are they running mod_inputs TAs?
Are they offloading parsing from the indexers?
How many intermediate forwarder deployments are there?
Which data sources are going to which intermediate deployment? How many EPS? Which source types? How much volume?

Understanding the full scope of what your heavy forwarders are doing today is critical to planning a successful migration.

Determining Splunk Edge Processor implementation feasibility

Question: Are your heavy forwarders acting as data collection nodes?

Considerations:

It’s sometimes the case that data collection workloads (mod_inputs) run on the same servers as intermediate routing and forwarding workloads.
Edge Processor does not support the role of a data collection node (DCN).
As a best practice, heavy forwarders should not run mixed workloads like this, but many environments have evolved this way over time.
If your heavy forwarders are performing data collection, these functions will need to be separated before or during migration.

Edge Processor does not do traditional parsing of events from Universal Forwarders. The parsing operation moves downstream to the next splunkd pipeline, usually to the indexing tier.

Considering Splunk Ingest Processor as an alternative

Question: Can Ingest Processor be used in addition to heavy forwarders for advanced data processing?

Considerations:

Ingest Processor can be used within Splunk Cloud Platform Victoria to perform processing of events while retaining upstream heavy forwarders for initial parsing.
Ingest Processor offers hosted Edge Processor functionality without changing heavy forwarder configurations.
This enables advanced data processing prior to indexing.
Retaining heavy forwarder infrastructure and moving parsing to Ingest Processor can simplify and reduce heavy forwarder operations while adding or maintaining transformations without additional infrastructure.

Assessing architecture complexity

Question: Does adding Edge Processor simplify your architecture or add unnecessary complexity?

Considerations:

Wholesale replacement of heavy forwarders with Edge Processor would, in most cases, introduce significant complexity.
Universal Forwarder-based data passing through the intermediate tier requires immediate remediation to avoid incorrect indexer parsing (see the Remediation Details section below).
There would have to be considerable benefit of using Edge Processor versus heavy forwarders to justify a 1:1 replacement.
Evaluate whether the benefits outweigh the migration effort and potential risks.

Evaluating incremental transition options

Question: Is an incremental transition possible?

Considerations:

If additional hardware can be provisioned, placing Edge Processor in-between the existing heavy forwarder tier and the destination can ease the migration burden. This approach keeps parsing at the Heavy Forwarder while enabling Edge Processor to further transform events.
Over time, the line breaking logic can be migrated to either Edge Processor or the destination as appropriate.

Incremental migration path


Phase	Routing	Processing
Initial	UF -> HF -> Index	All parsing and transformation done on HF.
Intermediate	UF -> HF -> EP -> Index	Line breaking and timestamping done at HF. Transformation and routing done at EP.
Target	UF -> EP -> Index	Line breaking and transformation done on EP. Indexer can optionally run TA transformations.

Identifying advanced processing needs

Question: Are there advanced processing needs that Edge Processor or Ingest Processor can fulfill better than heavy forwarders can?

Considerations:

Edge Processor/Ingest Processor and the use of SPL2 can more easily address "advanced" processing such as:
- Schema transformation
- Lookups
- Multiple routes
- Logs to metrics conversion
- Stats
For small changes like basic filtering and masking, heavy forwarders are fully capable of performing these functions.
Consider whether your processing needs justify the migration effort.

Making the final decision

Action: Collaborate with Splunk On-Demand Services, Solutions Architects, Theater Architects, and Professional Services to assess the level of effort for integrating Edge Processor or Ingest Processor.

Question: Based on the above evaluations, is Edge Processor or Ingest Processor integration the best path forward?

Considerations:

Weigh all pros and cons identified through the discovery process.
Factor in additional Edge Processor infrastructure requirements.
Consider additional licensing cost of Ingest Processor for volumes exceeding 500 GB/day.
Evaluate the total cost of ownership including migration effort, training, and ongoing operations.

Important note on routing protocols

The following guidance is specific to routing data to the Splunk platform. It is also assumed that forwarders and Edge Processor will be using Splunk protocol (s2s) to send that data.

Alternative protocols not covered in this guide:

Splunk forwarders have the ability to send data to Splunk through s2s over HTTP (httpout).
Edge Processor has the ability to send data to Splunk through HTTP Event Collector (HEC).

Using httpout or HEC can change the outcomes described in this guide dramatically and are not covered by this guide. Similarly, routing to other destinations (TCP, syslog, S3, NFS) are not covered by this guide.

Common migration scenarios

The following guidance details the most common intermediate routing architectures and provides context and high-level migration steps that will likely be encountered as part of a migration. In practice, there are many bespoke configurations that might not be represented below. This guide is not a runbook, and additional architectural and planning guidance will be needed for production migrations. You should almost always engage with your account team and Splunk On-Demand or Professional Services for migration guidance.

Scenario 1: Universal forwarder intermediate routing tier

Click here for details

Architecture: UF > IUF > Splunk platform

Current implementation details

Role of intermediate forwarder (IUF)

Lightweight, low-touch aggregation layer
Network firewall rule simplification
Can be used to simplify certificate management

Common configuration details

Splunk Cloud Platform Universal Forwarder package (certificates, server names)
Possibly non-default parallel pipelines configured for throughput

Infrastructure notes

Funneling precautions usually in place (except Victoria)
Typically follows a 2:1 forwarder to indexer ratio

Parsing role

Universal Forwarders usually do no parsing except in some rare configurations
Universal Forwarder events usually remain unparsed
Strictly passthrough in most cases
Indexer carries parsing burden

Configuration warnings

Sometimes event breaking and parsing is configured on the Universal Forwarder and should be considered during a migration.
Review all inputs.conf and props.conf configurations on intermediate Universal Forwarders.

Edge Processor migration details

Migration complexity

Low - This is the easiest of replacement scenarios.

Key migration points

If using as a pure passthrough/aggregation layer, the Edge Processor default pipeline and default configuration is acceptable.
The Splunk Cloud Platform Universal Forwarder package is used to configure Edge Processor by default.
Minimal configuration changes are typically required.

Pre-migration checklist

Verify that intermediate Universal Forwarders are not performing any parsing.
Review inputs.conf for any custom configurations.
Review props.conf for any event breaking rules.
Confirm network connectivity requirements for Edge Processor.
Plan for certificate management transition.

Scenario 2: Heavy forwarder Splunk intermediate routing tier

Click here for details

Architecture: UF > IHF > Splunk platform

Current implementation details

Role of intermediate forwarder (IHF)

Aggregation tier for consolidating data from multiple Universal Forwarders
Network firewall rule simplification
Can be used to simplify certificate management
Offload technology add-on (TA) workload from indexers

Common configuration details

TAs usually installed on heavy forwarders
Time or volume-based load balancing usually configured
Acknowledgment (ACK) can be configured on both input and output

Infrastructure notes

Funneling precautions usually in place (except Victoria)
Typically follows a 2:1 forwarder to indexer ratio
Significantly more hardware required than Universal Forwarder intermediate tier
Parallel pipelines often in use to take advantage of more powerful hardware

Parsing role

Events change from unparsed to parsed as they pass through the heavy forwarder
Props/transforms and rulesets (Ingest Actions) can be used here
This is often where significant data transformation occurs

Edge Processor migration details

Migration complexity

Moderate to high

Key migration steps

Scope configuration migration:
- Determine which configurations and/or TAs will move to the indexing tier.
- Identify which configurations will become Edge Processor pipelines.
- Document all current props/transforms configurations.
Technology add-on migration:
- For Splunkbase/supported TAs, consider migrating the TA to Splunk Cloud Platform.
- Verify TA compatibility with your Splunk Cloud Platform version.
- Test TA functionality after migration.
Custom configuration migration:
- For bespoke props/transforms, consider writing an Edge Processor pipeline.
- Translate transformation logic to SPL2.
- Test pipeline functionality thoroughly.
Line breaking configuration:
- Any source types routed through heavy forwarders must have Edge Processor line breaking defined.
- Review all source types currently processed by heavy forwarders.
- Create corresponding line breaking rules in Edge Processor.
Understand parsing changes:
- Timestamping and line breaking moves to the indexing tier.
- Events remain unparsed through Edge Processor.

Critical considerations


Consideration	Impact
Dormant transforms activation	Events passing through Heavy Forwarders become parsed, which means indexer TAs that use transforms don't have any effect on those events. Events that pass through Edge Processor will remain unparsed, so those "dormant" transforms may unexpectedly become active at the indexer layer. This can cause unexpected behavior and must be carefully evaluated.
Load balancing limitations	Edge Processor only supports time-based load balancing. If you are currently using volume-based load balancing, you might need to adjust your approach.
Acknowledgement not supported	Edge Processor does not support ACK on Splunk connections. If you rely on acknowledgment for data integrity verification, alternative approaches need to be considered.
Line breaking limitations	Edge Processor does not support every line breaking configuration available on heavy forwarders. Review your current line breaking rules for compatibility.

Pre-migration checklist

Inventory all TAs installed on heavy forwarders.
Document all props.conf configurations.
Document all transforms.conf configurations.
Identify all Ingest Actions rulesets.
List all source types processed by heavy forwarders.
Identify any volume-based load balancing configurations.
Document ACK configurations.
Review indexer-side transforms for potential activation.

Scenario 3: Heavy forwarder HEC routing tier

Click here for details

Architecture: HEC > Load Balancer > HF > Splunk platform

Current implementation details

Role of intermediate forwarder

Provide dedicated HEC processing endpoint
Network firewall rule simplification
Centralized HEC token management

Common configuration details

TAs usually installed on heavy forwarders
Time or volume-based load balancing usually configured

Infrastructure notes

For more than one Heavy Forwarder, a third-party load balancer is generally used to consolidate connections
Load balancer health checks configured against Heavy Forwarder HEC endpoints

Parsing role

HEC events are always parsed when received
Props/transforms and rulesets (Ingest Actions) can be used here

Configuration warnings

HEC tokens behave differently between heavy forwarders and Edge Processor
This difference is critical and must be understood before migration

Edge Processor migration details

Migration complexity

Moderate to high

Key migration steps

Review HEC token behavior:
- Carefully review Edge Processor HEC token behavior.
- Understand the differences from heavy forwarder token behavior.
- Plan for token migration or re-creation.
Resource planning:
- Ensure sufficient Edge Processor resources to meet HEC connection needs.
- HEC workloads can be connection-intensive.
- Plan for peak load scenarios.
Load balancer configuration:
- An external load balancer is almost always required for both heavy forwarders and Edge Processor when HEC is used.
- However, the configuration of that load balancer, especially service health checks, depends on Edge Processor token configuration.

Critical load balancer consideration

When tokens are enabled on Edge Processor, the HEC health endpoint must be retrieved with a valid HEC token. Not all load balancers can do this. Before migration, verify that your load balancer supports:

Passing authentication tokens in health check requests
Custom header configuration for health checks
Token-based health endpoint validation

Pre-migration checklist

Document all HEC tokens and their configurations.
Review Edge Processor HEC token documentation.
Assess Edge Processor resource requirements for HEC load.
Verify load balancer capabilities for token-based health checks.
Plan load balancer configuration changes.
Test health check configuration in non-production environment.

Scenario 4: Heavy forwarder syslog collection and routing tier

Click here for details

Architecture: Syslog > IHF > Splunk platform

Refer to the Edge Processor Validated Architecture: Syslog for further details on syslog collection best practices.

This section is specific to scenarios where the heavy forwarder is the syslog listener. For scenarios where syslog is collected by SC4S, syslog-ng, or other syslog tools, refer to Scenario 2 (Heavy Forwarder Splunk Intermediate Routing Tier) since the protocol/architecture will be Splunk (s2s) or HEC.

Current implementation details

Role of intermediate forwarder

Acting as the syslog listener and initial collection point of syslog network traffic (TCP/UDP)
First point of data ingestion for syslog sources

Common configuration details

One or more network port listeners configured
Initial sourcetyping and metadata assignment usually performed with TAs
Props/transforms for event parsing and field extraction

Infrastructure notes

Load balancing strategy for heavy forwarder syslog listeners will likely be the same for Edge Processor listeners
Consider network topology and firewall rules

Parsing role

Initial index, source, and source type set by input and props configurations
Line breaking defined by props/transforms
Heavy forwarders commonly perform significant parsing of syslog events

Configuration warnings

Heavy forwarders are Syslog RFC agnostic, whereas RFC/Port alignment in Edge Processor is crucial
Event host metadata logic is different between heavy forwarders and Edge Processor

Edge Processor migration details

High. As with the other scenarios above (and perhaps even more commonly), Heavy forwarders tend to do a lot of parsing of syslog events via props/transforms. Careful review of each heavy forwarder syslog input is required to properly migrate to Edge Processor. This can be a significant effort.

Migration complexity

High

Key migration points

Inventory syslog inputs:
- Document all syslog ports and protocols currently in use.
- Identify the RFC format of each syslog source.
- Map sourcetypes to syslog inputs.
RFC alignment:
- Edge Processor syslog ports need to align to source RFC.
- Reconfiguration of upstream syslog sources/ports might need to be modified to be compatible with Edge Processor.
Incorrect RFC assignment in Edge Processor will result in invalid event metadata.
Evaluate encrypted transport requirements:
- Heavy forwarders support syslog over TLS without mutual TLS (mTLS).
- Edge Processor only supports mTLS, which is not widely supported by syslog sources.
- This limitation might prevent some sources from directly migrating to Edge Processor.
Host Metadata Remediation:
- Ensure host metadata resulting from heavy forwarder processing matches Edge Processor processing.
- The logic used to derive the host between heavy forwarders and Edge Processor is different.
- This will very likely need remediation to maintain consistency.
Props/Transforms Review:
- Carefully review each heavy forwarder syslog input configuration.
- Translate parsing logic to Edge Processor pipelines.
- Test thoroughly to ensure event fidelity.

Critical considerations


Consideration	Impact
RFC alignment	Edge Processor requires explicit RFC configuration per port. Misalignment causes metadata corruption.
mTLS requirement	Many syslog sources do not support mTLS, potentially blocking direct migration for encrypted transport.
Host metadata differences	Host field derivation logic differs significantly and requires explicit remediation.
Parsing complexity	Syslog parsing configurations are often complex and require careful translation to SPL2.

Pre-migration checklist

Inventory all syslog inputs (ports, protocols, source counts).
Document RFC format for each syslog source.
Identify sources using encrypted transport (TLS).
Verify mTLS support for encrypted syslog sources.
Document current host metadata derivation logic.
Document all props.conf configurations for syslog source types.
Document all transforms.conf configurations for syslog source types.
Plan for sources that cannot migrate due to mTLS limitations.

Remediation details

Understanding why remediation is necessary is critical to planning a successful migration.

Why remediation is required

Splunk Edge Processor does not parse events in the same way that heavy forwarders parse events. Unparsed data that is sent to heavy forwarders undergoes line breaking and timestamping, whereas only linebreaking is performed in Edge Processor. Notably, Edge Processor does not set events to parsed/cooked as heavy forwarders do.

If heavy forwarders are removed from the existing processing flow, then the burden of parsing has to be handled by the downstream Splunk environment, which in many cases is not configured or prepared to handle that by default.

What remediation involves


Remediation task	Description
Source type migration	Source types and TAs that are defined on the heavy forwarders have to be migrated to Splunk Cloud Platform in most cases.
Line breaking rules	In most cases, the line breaking rules for those source types also have to be configured in Edge Processor.
Individual evaluation	An evaluation of each source type has to be made and individually remediated before a wholesale replacement can be made.

Remediation complexity assessment

While remediation may not be overly complex for any single source type, the cumulative effort across all source types can be significant. Consider:

Total number of source types processed by heavy forwarders
Complexity of props/transforms configurations per source type
Custom versus Splunkbase TA configurations
Testing requirements for each source type

Recommended approach

This is why an intermediate topology of UF → HF → EP → Splunk platform should be considered to ease the transition. The intermediate topology allows you to:

Maintain existing parsing on heavy forwarders.
Add Edge Processor transformations incrementally.
Migrate line breaking rules to Edge Processor over time.
Validate each change before proceeding.
Roll back specific changes if issues arise.

Alternative: Splunk Ingest Processor

If you prefer to retain your existing heavy forwarder infrastructure or want to add advanced processing capabilities without significant architectural changes, Ingest Processor might be an appropriate solution.

Ingest Processor benefits

Provides hosted Edge Processor functionality within Splunk Cloud Victoria
Does not require changes to existing Heavy Forwarder configurations
Enables advanced data processing prior to indexing
Simplifies and reduces heavy forwarder operational overhead
Adds or maintains transformations without additional on-premises infrastructure

Ingest Processor considerations

Available within most Splunk Cloud Platform Victoria environments
Licensing considerations apply for volumes exceeding 500 GB/day

When to consider Ingest Processor

You want to retain existing heavy forwarder infrastructure
You need advanced processing capabilities quickly
You want to reduce heavy forwarder complexity without full migration
Your environment is on Splunk platform Victoria
You value less overhead/maintenance over more control over your environment.

Comparing options: Heavy forwarder, Splunk Edge Processor, and Splunk Ingest Processor

When evaluating alternatives, you might also be considering third-party solutions. The following guidance can help you compare options.


Capability	Heavy forwarder	Edge Processor	Ingest Processor
Parsing	Yes	Limited	Yes
Transformation	Limited	Yes	Yes
Filtering	Yes	Yes	Yes
Masking	Yes	Yes	Yes
Routing	Yes	Yes	Yes
Schema transformation	Limited	Yes	Yes
Logs to metrics	Limited	No	Yes
Lookups	Limited	Yes	Yes
Data collection pull	Yes	No	No
UI-based management	No	Yes	Yes
Central config	Deployment server	Yes	Yes
Aggregations	No	Yes	Yes

Decision framework

Choose heavy forwarder when:

You need data collection capabilities on the same infrastructure.
Your transformation needs are basic (filtering, masking).
You prefer configuration file-based management.
You have existing heavy forwarder expertise.

Choose Edge Processor when:

You need advanced transformation capabilities.
You want centralized, UI-based configuration management.
You're willing to invest in migration effort or it’s a new data source entirely.
You want to leverage SPL2 capabilities.

Choose Ingest Processor when:

You want advanced processing without infrastructure changes.
You're on Splunk Cloud Victoria.
You want to retain heavy forwarder parsing.
You need quick time-to-value for advanced processing.

Third-Party solution considerations

If you are considering third-party solutions, compare specific use cases proposed by the vendor or desired by your organization and align them to heavy forwarder, Edge Processor, and Ingest Processor capabilities. Consider:

Total cost of ownership including licensing, infrastructure, and operations
Integration complexity with your existing Splunk environment
Support and maintenance requirements
Long-term strategic alignment with your data management goals

Next steps

Complete the migration discovery process.
- Use the evaluation questions in this guide to assess your environment.
- Document your findings and identify key considerations.
Document your current architecture.
- Inventory all intermediate forwarder configurations.
- Map data flows and processing logic.
- Identify dependencies and integration points.
Contact Your Splunk account team.
- Discuss your specific migration requirements.
- Understand licensing implications.
- Get connected with appropriate resources.
Engage Splunk Services.
- Work with Splunk On-Demand Services or Professional Services for architectural guidance.
- Develop a detailed migration plan.
- Establish success criteria and validation approach.
Plan for training.
- Identify team members who will manage Edge Processor.
- Plan for SPL2 training and enablement.
- Build internal expertise before migration.

Summary

Migrating from intermediate forwarders to Edge Processor can provide significant benefits including centralized configuration management, advanced transformation capabilities through SPL2, and improved processing efficiency. However, the migration requires careful planning and consideration of your specific environment.

Key takeaways:

Assess thoroughly: Use the discovery questions to understand your current state and requirements
Consider complexity: A wholesale replacement often introduces significant complexity; evaluate incremental approaches
Understand parsing changes: Edge Processor does not parse events the same way heavy forwarders do; plan for remediation
Leverage expertise: Engage Splunk services to help plan and complete your migration
Consider alternatives: Ingest Processor might provide benefits without full migration effort

For questions about your specific migration scenario, please contact your Splunk account team or Splunk Support.

Additional resources

These additional Splunk resources might help you understand and implement this guidance:

Splunk Help: Edge Processor Validated Architecture: Syslog
Splunk Help: How the Edge Processor solution transforms data
Splunk Help: Install and configure the Splunk Cloud Platform universal forwarder credentials package
Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their Success Plan. Engage the ODS team at ondemand@cisco.com if you would like assistance.