Skip to main content
Lantern Home

 

Splunk Lantern

Limiting concurrent searches

  1. Last updated
  2. Save as PDF

 

While the Splunk platform is designed to handle vast amounts of data and deliver insights at speed, resources are not infinite. Without careful oversight, concurrent searches can rapidly consume available resources, leading to a noticeable slowdown in response times or even system instability. This section explains the significance of regulating concurrent searches and underscores how unmanaged searches can adversely impact the efficiency and overall performance of the Splunk platform.

This section outlines the following steps in limiting concurrent searches:

  1. Understanding concurrent searches
  2. Assessing the risks of unrestricted concurrent searches
  3. Setting appropriate limits on concurrent searches
  4. Implementing best practices for efficient search management
  5. Monitoring and adjusting concurrent search limits

Understanding concurrent searches

The Splunk platform has powerful search capabilities that enable users to sift through vast datasets to extract meaningful insights. At the core of this capability is the concept of concurrent searches. But what exactly does this mean? Concurrent searches refer to multiple search queries executed simultaneously within your Splunk environment. Instead of processing searches sequentially, which would take a significant amount of time. The system is designed to handle several queries at once to maximize the utilization of resources.

To better understand how concurrent users and searches impact performance, see Scale your Splunk Enterprise deployment on Splunk Docs.

The characteristics of concurrent searches can vary based on the type of search (real-time vs historical), the complexity of the search, and the amount of data being queried. Some searches, due to their complexity or the sheer volume of data they encompass, might be resource-intensive, while others might be lightweight and quick.

For Splunk performance optimization, you need to prioritize the management of these concurrent searches. Just as a highway can only handle so many cars at once before traffic jams occur, the Splunk platform has a finite amount of resources to process searches. If too many heavy, resource-intensive searches run simultaneously, it can lead to bottlenecks. This can manifest as slower search return times, reduced system responsiveness, or even timeouts. That makes understanding and effectively managing concurrent searches not just a good-to-know concept, but a good skill for maintaining the health and efficiency of your Splunk environment.

Assessing the risks of unrestricted concurrent searches

Search capability in the Splunk platform is one of its most key features. However, when mismanaged, this strength can quickly become a liability. Allowing unrestricted concurrent searches can lead to a series of adverse effects on the system, users, and the overall reliability of the Splunk platform. Here are a few examples of issues that could be experienced:

  • Resource Depletion: Every search in the Splunk platform consumes system resources - CPU, memory, and I/O operations. When too many searches run simultaneously, especially if they are complex or span extensive datasets, there's a substantial risk of overwhelming the system. This can lead to slowed operations, lagging user interfaces, or even system crashes.
  • Degraded Search Performance: The Splunk platform is optimized to return results as quickly as possible. But if it's juggling too many concurrent searches, the processing power for each individual search is reduced. The result? Slower search times and, in extreme cases, search timeouts. For users, this can be frustrating, especially if timely insights are needed.
  • Increased Queue Times: The Splunk platform uses a queuing system to manage searches, especially during high-load periods. If unrestricted searches are allowed, the queue can become backlogged, meaning even simple searches could be delayed because they're waiting for resources to free up.
  • Reduced Data Integrity: With overwhelming search loads, there's a potential for incomplete or even incorrect data returns. For instance, if a search is terminated prematurely due to resource constraints, users might not get the full picture of their queried data.
  • Operational Inefficiencies: In enterprise settings, the Splunk platform often plays a pivotal role in decision-making processes. If searches are consistently slow, unreliable or completely absent due to unrestricted concurrency, it can introduce operational inefficiencies. Decisions might be delayed, or worse, made on incomplete data, even missed altogether.
  • Strain on Infrastructure: Continual maxing out of system resources can put a strain on the underlying infrastructure, leading to wear and tear on physical components or increased cloud costs, if hosted in a cloud environment.

While the Splunk platform has built-in mechanisms to manage concurrent searches, relying solely on defaults without understanding and fine-tuning based on the specific needs of an organization can be a risky approach. The goal should always be to strike a balance: ensuring users can get the data they need when they need it, without compromising the health and performance of the Splunk system.

Setting appropriate limits on concurrent searches

Ensuring optimal performance and preventing system overloads in the Splunk platform necessitates a cautious approach to managing concurrent searches. By setting appropriate limits, you not only maintain system responsiveness but also ensure that users can extract the data they need without undue delays. The needs of limits tend to be dynamic and can differ from one system to another. Further detail of the main assessment criteria and methods of application has been captured below.

Determining optimal concurrent searches for your environment

Before imposing any limits, take a some time to assess your environment. Consider the number of users, the complexity of typical searches, and the frequency of those searches. A small team running simple searches might require fewer concurrent search limits compared to a larger analytics-driven organization.

Factoring in system resources and user demands

Your system's CPU, memory, and I/O capabilities play a pivotal role in determining search limits. A more robust system might handle more concurrent searches than a resource-constrained one. Furthermore, gauge your user base: do they run intensive ad-hoc searches frequently, or are they more into scheduled, routine searches?

Example: If you have a quad-core CPU with 16GB RAM dedicated to the Splunk platform and typically notice only a fraction of this being used, you might consider allowing a higher number of concurrent searches. However, if you're frequently maxing out resources, it might be time to be stricter with your limits or consider a system upgrade.

Configuring search limits in the Splunk platform

The Splunk platform provides two primary mechanisms for setting search concurrency:

  1. Role-Based Search Concurrency: This allows you to set search limits based on user roles. For instance, you might allow administrators to run more concurrent searches than regular users, given their elevated requirements. To configure role-based concurrency:
    1. Edit the authorize.conf file located in $SPLUNK_HOME/etc/myapp/local/.
    2. Find the specific role.
    3. Adjust the value under the cumulativeSrchJobsQuota and srchJobsQuota section to desired setting.

    Example: For an analytics-heavy role, you might allow up to 10 concurrent searches, while a regular user role might be limited to three.

  2. System-Wide Search Concurrency: This sets a blanket limit across the entire Splunk environment, regardless of user role. To configure system-wide concurrency:
    1. Edit the limits.conf file located in $SPLUNK_HOME/etc/myapp/local/.
    2. Modify or add the [search] stanza with max_searches_per_cpu and base_max_searches attributes.

    Example:

    [search]
max_searches_per_cpu = 4
base_max_searches = 6

    This configuration allows four searches per CPU core with a minimum of six searches regardless of CPU core count.

For detailed steps and considerations, see Splunk official documentation on concurrent search limits.

Implementing best practices for efficient search management

Effectively managing searches in the Splunk platform ensures optimal system performance and achieving timely data retrieval. As your Splunk environment scales and user demands increase, instituting a set of best practices can make a significant difference in maintaining system efficiency.

Prioritizing critical searches

Identifying and prioritizing critical searches ensures that mission-critical operations have the resources they require. This can be accomplished by:

  • Setting Scheduled Search Priority: Use the scheduled search priority parameter in searches to prioritize them, with higher numbers indicating more importance.
  • Dedicated Search Heads: For extremely critical searches or dashboards, consider dedicating specific search heads. This ensures that vital operations are insulated from routine search activities.

Scheduling routine searches during off-peak times

Regular, routine searches can be resource-intensive, especially when executed frequently or across vast data sets. To minimize the impact on system performance:

  • Use Scheduled Searches: Schedule routine searches to run during times when the system faces lower user demand, like late nights or early mornings.
  • Leverage Report Acceleration: For recurring searches, enabling report acceleration can significantly speed up search execution by caching the results.

Educating users on efficient search practices and guidelines

End-users play a critical role in ensuring search efficiency. Providing them with guidance can lead to more effective search operations:

  • Training Sessions: Hold regular training sessions to educate users on efficient search syntax, use of wildcards, time frames, and other Splunk-specific functionalities.
  • Documentation: Create and distribute user-friendly documentation that provides tips, best practices, and examples of efficient searches.
  • Feedback Mechanism: Establish a feedback loop where users can report slow searches or seek optimization advice. This promotes continuous learning and optimization.

Monitoring and adjusting concurrent search limits

Efficiently managing performance of the Splunk platform involves not only setting concurrent search limits, but also regularly monitoring and adjusting them based on evolving system needs and business requirements. Effective monitoring helps in preempting potential issues, ensuring smooth system operations, and optimizing resource allocation. Here's a closer look at the facets of this process:

Tools and metrics for monitoring search performance and concurrency

The Splunk platform provides some tools and metrics that enable tracking of search performance and concurrency.

  • Monitoring Console: The Splunk Monitoring Console offers a consolidated view of the system's health and performance metrics, including those related to search concurrency.
    • Navigate to: Settings > Monitoring Console > Search > Activity > Instance. This dashboard provides a real-time overview of active, queued, and skipped searches, aiding in quick diagnosis and action.
  • Search Job Inspector: For detailed insights into individual search performance, use the Search Job Inspector. It reveals specifics about the search's execution, including time taken, scanned events, and used resources.
  • Metrics Log: The metrics log contains granular data about the Splunk platform operations. Queries targeting this log can discover insights into search concurrency, performance bottlenecks, and more.

Regularly reviewing and adjusting search concurrency settings

Static configurations rarely meet the dynamic demands of a growing Splunk environment. Regular reviews ensure that search concurrency settings remain aligned with the system's capabilities and your organization's needs.

  • Assess System Load: Regularly monitor the system's CPU, memory, and I/O performance. If the system is consistently under heavy load during peak search times, consider adjusting search limits.
  • Factor in New Business Requirements: As your organization grows and use cases evolve, search demands might change. Maybe there are more users, or perhaps certain new searches have become business-critical. Regularly reassess and realign search concurrency limits in light of these changes.
  • Engage with End-Users: Maintain open communication channels with Splunk end-users. Their feedback about search performance, delays, or system responsiveness can be pivotal in making informed adjustments.
  • Regularly Review Implemented Use-Cases: Just as there is a need to monitor the load to cater for new use cases. Reviewing business requirements and engaging with end-users is also necessary to ensure that any implemented use-cases are regularly reassessed and validated so as to not reduce resource efficiency.

Helpful resources

This article is part of the Splunk Outcome Path, Reducing search load. Click into that path to find ways to reduce search load and better allocate resources to lead to a highly efficient and cost-effective Splunk environment.

In addition, these resources might help you implement the guidance provided in this article:

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you would like assistance.