Skip to main content
Lantern Home

 

Splunk Lantern

Improving performance in Enterprise Security 8

  1. Last updated
  2. Save as PDF

 

In today’s security operations, every second counts. With growing data volumes and increasingly complex environments, slower key performance indicators can mean the difference between stopping an intrusion and losing critical ground. Intruders can move laterally within your network in just minutes. If your detection pipeline lags because searches or dashboards load slowly, your analysts are already behind.

Faster page loads and searches translate directly to faster analyst action. Splunk Enterprise Security (ES) performance is built on multiple layers: backend Python processing, SPL search execution, frontend JavaScript, KVStore interactions, web server performance, and the underlying search infrastructure.

Improvements at the Splunk platform level—such as enhanced search infrastructure, data indexing, and API performance—ripple upward to benefit ES automatically. Conversely, when ES reduces load through smarter searches, the platform runs more efficiently.

To maximize these benefits, here’s how you can optimize your environment across four key pillars: Experience, Velocity, Efficiency, and Resilience.

Solution

Experience: How analysts interact with pages in Enterprise Security

Compared to previous versions, Splunk Enterprise Security 8 provides two times faster API calls, 25 percent faster loading of the user interface, and over 50 percent improvements in loading the analyst queue.

What can you do to maintain experience gains in your environment?

  • Keep duplicated search jobs to a minimum. Every search that you eliminate is capacity that can be used elsewhere.
  • Stick to a single analyst queue tab where possible.

    Starting with ES 8.2, search ID caching reuses existing search job IDs across tabs, delivering up to 50 times faster repeated query results.

  • Apply search best practices, such as filtering by user, time, status, severity, or other filters.
  • Be cautious when modifying the core analyst queue searches as improper changes can downgrade performance.

Velocity: How quickly you can move from a question to an answer

Recent updates to Splunk Enterprise Security and the Splunk platform have optimized velocity in the following ways:

  • API optimizations to over 13 separate pages to reduce prop loads to only those that are required, cutting down on processing and transfer time. Most pages saw a 100 percent decrease in load time.
  • Custom search commands run on the search head, which can become a performance bottleneck. Macros, on the contrary, can run on the indexer as part of distributed search. During benchmarking, macros were found to be 15 times faster compared to the custom commands on a standalone deployment, so using more macros improves velocity.
  • The team reviewed enrichment lookups in ES and eliminated five that weren’t necessary. They also removed ten joins. Those changes alone cut page load times by about 30 percent. This sped up investigations, the analyst queue main table, side panels, the risk event timeline, AI Assistant, and bulk update workflows.
  • Enhancements led to SmartStore fetch and indexing throughput improvements.

What can you do to maintain velocity gains in your environment?

  • Push execution as close to the data as possible. Prefer distributable SPL commands that run on indexers early in the pipeline.
  • Avoid bringing large datasets back to the search head, which limits parallelism and increased memory use. 
  • Use search head centralized commands sparingly and intentionally. 

Efficiency: How to make the best use of available resources

Splunk Enterprise Security 8 supports KVStore retention based on size, time, or a combination, allowing automatic pruning of stale records. A leaner KVStore means faster lookups and reduced system load.

We’ve improved how scheduled searches are distributed to avoid resource contention. Instead of all scheduled searches running at set intervals, if we can space them out, we get much better performance and a healthier resource utilization profile across indexers. The allow_skew option lets searches run within a defined window rather than all firing at the same time. ES 8.2 automates this leveling, improving performance consistency.

Not all cron schedules are skewable.

What can you do to maintain efficiency gains in your environment?

  • Prune lookup tables.
  • In a pre-8.2 environment, you can reduce concurrency spikes by turning on allow_skew and hand-tuning your search cron schedules.
  • If allow_skew is not enabled yet, schedule searches to spread the load over time.
  • Review your stack sizing and ensure it is a good fit for your workload.
  • For data model accelerations, optimize SPL so you only scan what you need. Be specific about indexes, source types, and tags. Unoptimized DMA searches with (index=* OR index=_*) clause that are not allowlisted with the specific indexes are problematic because they consume most of the system resources on the indexers. Extremely slow DMA searches can have up to four concurrent searches, which significantly limits available vCPU cores on the indexers for the remaining searches.
  • Custom commands in DMA search can have unwanted consequences, so try to avoid them.
  • Use ingest actions to route unneeded logs away from DMAs, reducing scanned data volume.

Resilience: How to keep your systems available and responsive under stress

Improvements we've made here include enhanced workload management, tighter cross-product integrations, Splunk Classic replication, and cross-region disaster recovery in Splunk Cloud Platform. These are the safety nets that ensure performance gains don’t disappear the moment you hit an unexpected load spike.

What can you do to maintain resilience gains in your environment?

  • Use workload management to quarantine resource-heavy ad-hoc searches. Note that this is not the same thing as managing search concurrency. Workload management makes sure the noisy jobs wait in line while the important ones get right in.
  • If you’re on Splunk Cloud Platform, ensure you’re on the Victoria Experience for higher scale limits.
  • For on-premises environments, make sure your cluster manager is tuned and your stack is appropriately sized.

Next steps

Now that you have an idea of how Splunk Enterprise Security 8 can improve your security operations, watch the full .Conf25 Talk, Blazing-fast security ops: Unleashing Splunk ES 8.0 for speed and scale. In the talk, you'll see more specific examples of these principles, learn about results other customers have had from following these steps, and learn about more upcoming improvements.

In addition, you might find these Splunk resources helpful:

  • Written by Bhanu Karumuri and Brent Davis
  • Software Engineers at Splunk