Skip to main content
Os artigos do Splunk Lantern estão agora disponíveis em português.
 
 
 
Splunk Lantern

Using Edge Processor to save Splunk Virtual Compute

 

Splunk Virtual Compute (SVC) utilization is a measurement of the resources that are employed by the Splunk stack, as well as the measurement by which customers pay for their Splunk instance. The more efficient a stack is, the more value you will get out of each SVC and thus, your Splunk deployment. In this document, we elaborate on how Splunk Edge Processor can be used to replace Data Model Acceleration (DMA) processes in order to reduce SVC utilization. 

Background

Splunk Edge Processor (EP) is a data processing solution that works at the edge of your network and can be used to filter, mask, and route your data close to its source before it gets to the Splunk platform or other external environments. For example, EP can be used to create indexed fields from the raw data. Indexed fields are indexed in the Splunk platform upon ingestion, rather than evaluated at search time. Such fields can improve the efficiency of compute-heavy Splunk processes which in turn, can contribute to a reduction in the overall SVC usage.

One such compute-heavy process is Data Model Acceleration (DMA). DMA is used by the Common Information Model (CIM) application, which is used by Splunk Enterprise Security (ES) but can also be used independently. DMA indexes CIM data which enables faster search performance on this data. In effect, DMA runs SPL queries periodically in the background to index CIM data and if these queries are complex or the dataset is large, DMA can consume a considerable amount of compute power. In this article, we show how Splunk Edge Processor can be used to extract indexed CIM fields and thus eliminate the need for the DMA process. In turn, this can reduce the amount of SVCs consumed.

In the following section we explore how, by creating two different test settings (with and without Edge Processor), you can assess the amount of SVC usage that can be reduced.

Test settings

In order to measure the amount of potential SVC savings that can be gained by eliminating the DMA process we created the following setups:

  • Setup 1 (with DMA, without Edge Processor): Data is sent directly to an indexer cluster with the Splunk Add-on for Microsoft Windows installed on the Search Head. DMA is scheduled to run every 5 minutes, accelerating the 19 default models.
  • Setup 2 (without DMA, with Edge Processor): Data is sent to two Edge Processor instances that extract the CIM fields. The raw event and the extracted fields are sent to the same indexer cluster as in Step 1. Since the CIM fields are already extracted, the DMA process is unnecessary and is turned off. 

We used 34 Windows events with source type WinEventLog:Security.

Results

Setup Number of accelerated models Throughput (Events/sec) Indexer throughput (Events/sec) Total SVC usage SVC consumers
Edge Processor CIM field extraction N/A 20k 20k 11.18 ingest 10.07
search 1.04
shared 0.07
Accelerated data models 19 20k 20k 19.43 ingest 10.00
search 8.04
shared 1.38

Conclusion

In our test, Edge Processor saved 42% of the SVCs compared to the DMA process. As expected most of these savings came from the search process, which is unnecessary when using Edge Processor to extract the CIM fields. 

It is worthwhile to note that only the Windows TA was installed on the system when DMA was run. In a typical Splunk deployment, dozens and sometimes hundreds of TAs are installed. This configuration makes the search query that DMA executes much more complicated and therefore requires much more processing power and SVCs to compute. Thus, the benefits of using Edge Processor in such a scenario can be even greater. 

On the other hand, there are costs associated with running the Edge Processor nodes, such as the cost of the cloud or other physical instances on which the EP nodes run. These costs will vary from company to company, as will the costs of the saved SVCs. It is important to take all of this into account when considering the architecture that is right for you.  

Finally, your results will vary. The test described above is not meant to be comprehensive or to take into account all the variables associated with a real world Splunk deployment. It is only intended to suggest a possible way that may be effective in saving SVCs. We recommend you conduct a similar test in your environment, with your data and add-ons, to determine a more accurate estimation of the SVCs that you can save with Edge Processor.  

Next steps

To get access to Splunk Edge Processor, email edgeprocessor@splunk.com or reach out to your account team.

In addition, these Splunk resources will help you better understand SVCs and Edge Processor.

 

  • Written by Chandrima Sarkar, Doron Keller and Felix Jiang
  • Splunk