Skip to main content
 
 
Splunk Lantern

Reducing search load

 

Significant reductions in search load and better resource allocation lead to a highly efficient and cost-effective Splunk environment. There are many strategies you can apply to optimize search load and maximize SVC/vCPU utilization. These include refining search queries and dashboard panels, leveraging summary indexing to pre-aggregate data, implementing search-time filters, and using data model accelerations (DMA). You can also control time ranges, more effectively manage indexes, limit concurrent searches, and employ search head clustering. Lastly, you should regularly monitor of resource usage and keep your Splunk deployment up-to-date. The strategies provided in this pathway will help you accomplish all these varied goals. You can work through them sequentially or in any order that suits your current level of progress in reducing search load.

This article is part of the Reduce Costs Outcome. For additional pathways to help you succeed with this outcome, click here to see the Reduce Costs overview.

Optimizing searches and dashboards

Efficient search and reporting capabilities help you derive valuable insights from your data while minimizing costs. Optimizing searches and dashboards in the Splunk platform not only improves performance but also contributes to reducing the total cost of ownership (TCO) by optimizing resource utilization and maximizing productivity.

►Click here to read more.

In this section, you will explore how the following practices can lead to significant TCO savings:

  1. Refactoring search queries
  2. Leveraging search commands
  3. Optimizing search jobs
  4. Optimizing dashboards

Refactoring search queries

Optimizing search queries directly impacts TCO by reducing resource consumption and enhancing performance.

  • Reduced Data Processing: By assigning specific time ranges through role permissions, you can control and limit the scope of data users can access and search. As a result, costs associated with processing and storage are reduced, and users are provided with a more focused and relevant dataset based on their roles.
  • Positive Matching: Using positive matching to filter data reduces the amount of data retrieved from the indexers. While it is possible to filter data using negative arguments (for example, NOT), results are only filtered when data is collected from the indexers which results in inefficient searching. This can also affect search results.
  • Efficient Resource Utilization: Filtering data early in the search pipeline minimizes the amount of data processed by subsequent search commands. This reduces the strain on system resources, leading to improved performance and lower infrastructure costs.
  • Streamlined Data Analysis:
    • Aggregation commands such as stats and chart help consolidate and summarize data, reducing the volume of data processed. This optimization results in faster search execution, reducing the need for extensive computational resources and lowering infrastructure costs.
    • MapReduce allows for the efficient segregation of large volumes of data across various nodes (the 'map' phase), followed by the aggregation and synthesis of this data to derive meaningful insights (the 'reduce' phase). By distributing the computational load, MapReduce ensures that the Splunk platform can sift through logs and datasets effectively.

For specific guidance on refactoring search queries, see Optimizing search.

Leveraging search commands

Utilizing Splunk's search commands efficiently enhances search performance and contributes to TCO reduction.

  • Optimized Field Manipulation: Leveraging eval commands for field calculations and formatting improves data analysis efficiency. By preparing data for analysis during the search phase, subsequent processing steps can be simplified, reducing overall resource consumption and lowering TCO.
  • Streamlined Data Analysis: Aggregation commands such as stats and chart help consolidate and summarize data, reducing the volume of data processed. This optimization results in faster search execution, reducing the need for extensive computational resources and lowering infrastructure costs.

Optimizing search jobs

Efficient search job management minimizes resource waste and contributes to TCO reduction.

  • Controlled Result Sizes: Limiting the number of search results using commands like head or top reduces memory consumption and improves search performance. By managing result sizes, you can optimize infrastructure costs while still obtaining the necessary insights from the data.
  • Workload Management: Workload Management can be used to ensure the most important jobs are not resource constrained. For more information, see Workload management in the Reducing your infrastructure footprint pathway.
  • Identifying Unnecessary Scheduled Saved Searches: Identifying and cleaning up unnecessary scheduled saved searches, especially out-of-the-box (OOTB) scheduled saved searches that aren't required, streamlines the system and trims infrastructure costs. This includes removing redundant or unused saved searches in applications like Splunk Enterprise Security (ES) or Splunk ITSI (ITSI).
  • Resource Cleanup: Properly managing search job lifecycles, including canceling or terminating unnecessary or long-running jobs, prevents resource waste and optimizes system performance. This optimization reduces infrastructure costs by eliminating unnecessary resource usage.
    • Monitoring Console or the Cloud Monitoring Console can aid in this review process by providing at-a-glance insights into system health and performance. Regularly optimize, update, or remove searches based on changing needs.
    • Search Job Monitoring: Regularly monitor the ongoing search jobs in your Splunk environment. You can do this through the Splunk Search Job Inspector, which provides insights into active and historical search jobs.
    • Scheduled Searches Review: Examine your scheduled searches and reports. Determine whether all scheduled searches are still relevant and producing valuable insights. If there are reports that are rarely accessed or no longer provide significant value, consider discontinuing or optimizing them.
    • Stuck or Abandoned Jobs: Keep an eye out for search jobs that are stuck, running indefinitely, or have been abandoned. Canceling or terminating these jobs can free up resources.
    • Audit Search Usage: Review the usage and popularity of saved searches and reports. If certain searches are hardly ever used by users or teams, they might be candidates for optimization or removal.
    • Regular Review: Conduct periodic reviews of your search jobs to ensure they align with your organization's goals and requirements. Regularly optimize, update, or remove searches based on changing needs.

For specific guidance on optimizing search jobs, see Optimizing search.

The optimizations discussed here result in lower infrastructure costs, reduced hardware requirements, improved operational efficiency, and increased productivity. By embracing these best practices, you can extract maximum value from your Splunk deployment while minimizing total cost of ownership, ultimately leading to a more cost-effective and efficient data analysis environment.

Optimizing dashboards

Well-optimized dashboards not only improve user experience but also contribute to TCO reduction.

  • Resource Optimization: Consolidating multiple panels into a single panel reduces resource consumption and enhances dashboard loading times. This optimization translates to lower hardware requirements, reducing infrastructure costs.
  • Base and Chain Searches: Base and chain searches encapsulate common search logic and filters, which can be reused across multiple searches and dashboards. This approach minimizes redundant code, improves consistency, and simplifies maintenance by executing a single search and reusing the returned data through other sub-searches. By leveraging base and chain searches, you reduce development time and effort, optimize resource utilization, and enhance performance.
  • Accelerated Data Models: Utilizing data model acceleration or reducing the data set searched improves search and dashboard performance. Faster rendering and reduced computational demands result in lower resource consumption, leading to cost savings in terms of infrastructure and operational efficiency.
  • Efficient Data Access: By leveraging summary indexing for frequently used searches, dashboards can load faster and require fewer computational resources. This optimization minimizes the need for extensive data retrieval, reducing infrastructure costs and improving overall dashboard performance.

For specific guidance on optimizing dashboards, see Following best practices for working with dashboards.

Implementing search filters

Searching data in the Splunk platform requires precision and efficiency. Indiscriminate searches can consume system resources and extend processing times, leading to less than optimal experiences. Thankfully, the Splunk platform offers index-time and search-time filters to refine searches and ensure that users can focus on the most relevant data subsets without unnecessary overhead. This section offers a technical deep dive into these filtering methodologies, providing a structured approach to optimizing searches.

►Click here to read more.

This section outlines the following steps in creating search filters:

  1. Search filter basics
  2. Types of filtering
  3. Tips and best practices for filtering

If you are new to the Splunk platform or could use a search refresher before reading further into this topic, read Get started with search in Splunk Docs.

Search filter basics

In the Splunk platform, search filters act as a toolset for users navigating vast datasets. At its core, a search filter serves as a defined precedent or set of criteria that selectively filters data during a Splunk search operation. Think of it as a sifter, allowing only data that meets specific conditions to pass through, while other data remains excluded from the search results. To see a useful presentation on how Splunk search works, watch this .conf presentation: Behind the magnifying glass: How search works.

One of the most immediate benefits of utilizing search filters is the noticeable reduction in processing time. For instance, imagine searching through an extensive log repository for errors that occurred within the last 24 hours. Without filters, the Splunk platform would have to comb through perhaps months or even years of data. However, by applying a time-based filter, Splunk will focus only on the logs from the desired time frame, drastically cutting down the search's duration.

This time efficiency also translates into system load benefits. Without filtering, the Splunk platform would engage more system resources, including CPU cycles and memory, to process and display unnecessary data points. With optimized searches, there's less strain on system resources, ensuring the stability and responsiveness of the Splunk environment. Also, with strategic filtering in place, the Splunk platform only processes the essential, relevant chunks of data. For example, when looking for specific error codes, instead of searching every entry, the Splunk platform can be directed to only explore logs containing the term "ERROR" and the specific code, for example "E12345", significantly reducing the computational overhead.

Types of filtering

Search capabilities in the Splunk platform are attributable to its filtering mechanisms. While the broader concept of filtering might be familiar to most Splunk users, understanding the specific types of filters the Splunk platform offers is important for efficient data querying and analytics. The primary filter categories are index-time filters and search-time filters.

Index-time filters

  • What are index-time filters? Index-time filters are applied as data is being ingested into the Splunk platform. This means that data is filtered before it's even indexed, so only the data that meets the specific criteria makes it into the Splunk index. This data can either be routed to another index or sent to the nullqueue.
  • When and why to use them: Index-time filters are especially useful when there's a need to exclude specific data from being stored in the Splunk platform to begin with, either for privacy reasons, regulatory compliance, or simply to optimize storage. For instance, you might exclude verbose logs from a particular system or sensor readings from a malfunctioning device, ensuring they don't consume valuable storage space or clutter up search results.
  • Implement index-time filters: Before diving into the technical configurations, first decide which data to target with index-time filters. It's a strategic step: the goal is to identify data that, while perhaps available for ingestion, does not provide value when analyzed or stored. This might be because of redundancy, irrelevance, or even compliance mandates. Consider the following:
    • Are there log entries from a particular system or application that are excessively verbose and not useful for analysis?
    • Is there sensitive information that should never be stored in the Splunk platform for compliance or security reasons?
    • Are there event types or log sources that are irrelevant to the objectives of your Splunk deployment?
  • Configuring index-time filters: After you have a clear idea of the data you'd like to filter out at index-time, the next step involves configuration. This is done using props.conf and transforms.conf configuration files. In props.conf, you'll specify the data source and direct it to a transformation in transforms.conf, which then performs the actual filtering. Here is an example process to follow:
    1. In props.conf, identify the data source using a stanza, then associate it with a transformation.
    2. In transforms.conf, define the transformation, specifying the filtering criteria.
    3. Ensure you carefully test any index-time filter configurations in a staging environment first, as these filters can't be undone without reindexing the data.
  • Common examples and use cases
    • Redundant Data Removal: If you have a system that logs both a detailed and a summary event for every occurrence, but only the summary is relevant, an index-time filter can exclude the detailed logs.
    • Compliance-based Filtering: For regulatory reasons, you might want to exclude logs that contain sensitive personal information.
    • Infrastructure Noise Reduction: If certain machines or applications are known to produce "noisy" logs that don't contribute to meaningful analysis, these can be filtered at index-time.

By configuring index-time filtering, Splunk users can maintain a cleaner, more efficient, and more relevant dataset, ensuring that the system is not bogged down by unnecessary data and that storage costs are optimized. For more information on this topic, refer to Route and filter data on Splunk Docs.

Search-time filters

  • What are search-time filters? Search-time filters are applied when querying already indexed data. Instead of pre-filtering data during ingestion, this filtering type sifts through existing indexed data, selecting only the subsets of data that align with the defined criteria for the search.
  • Their advantages and use cases: Search-time filters provide the flexibility to dynamically refine search results without altering the underlying indexed data. They're invaluable when dealing with broad datasets where different queries might require different data subsets. For instance, an administrator could focus on logs from a particular server during a specific hour, or a security analyst might narrow down logs to specific IP addresses when investigating potential threats. Filtering types in the Splunk platform are tailored to address varied requirements, ranging from storage optimization with index-time filters to dynamic query refinement using search-time filters. Becoming better using these filters is a key competency for any Splunk user aiming for efficient and targeted data analytics.
  • Implement search-time filters: Search-time filtering operates on the principle of refining the data you've already indexed when conducting a search, instead of during the ingestion process. As such, the first step is to determine the subsets of data that are pertinent to your current search objectives. It's worth noting that the choice to apply search-time filters can be situational, often guided by the specific task at hand. Consider the following:
    • What specific data points or events are you looking to analyze?
    • Are there indexed datasets that, while potentially useful in other scenarios, are extraneous to your current search?
  • Configuring search-time filters: After identifying the desired data subsets, you'll transition to the Splunk search or the SPL (Splunk Processing Language) to implement the search-time filters. Using SPL, you can leverage various commands and functions, like search, where, and eval, among others, to narrow down the search results. For example, to filter events from a specific source or source type, you'd use a query like: source="your_source". Always ensure that your filter criteria are both precise and optimized for efficiency, as vague or overly broad search criteria can consume unnecessary system resources.
  • Common examples and use cases
    • Security Monitoring: If you're monitoring for security events but want to exclude routine login and logout events, a search-time filter like NOT (eventtype IN ("login","logout")) could be applied.
    • Operational Metrics: When tracking the uptime of a server, irrelevant events like user logins or software updates could be filtered out, focusing solely on start-up and shutdown events.
    • Debugging: If you're diagnosing a system error, you might apply search-time filters to exclude all events except those flagged as 'error' or 'critical'.

Implementing search-time filters effectively is a valuable skill for Splunk users. It enables them to create tailored searches that address specific analytical requirements without sifting through mountains of unrelated data.

If you looking for help writing better searches in the Splunk platform and applying the concepts of this article, Writing better queries in Splunk Search Processing Language has a lot of great, actionable tips.

Tips and best practices for filtering

Filtering in the Splunk platform, whether at index-time or search-time, streams data analysis and enhances system performance. However, striking the right balance between efficiency and comprehensiveness can be tricky. Here are some expert tips and best practices to ensure optimal use of filters in the Splunk platform.

Time is the most important search filter

Data stored in indexes is typically stored in buckets which are time bound. The most efficient filter that can be applied to searches is that of time as it will reduce the number of events that need to be read across the indexes and eventually processed. For data stored in SmartStore indexes, this is even more important as older buckets are stored in the cloud. These buckets are automatically copied to a local cache whenever they are required for search purposes.

Ensure efficient filtering without losing critical data

When crafting filters, specificity is paramount. While it's tempting to craft broad filters for convenience, doing so might inadvertently screen out crucial data points. Always verify the impact of your filters, especially after initial implementation.

Specify what you want, rather than what you don’t want

Using positive filters to specify what you want will increase the efficiency of your searches because the system will be able to apply those filters directly to the data that needs to be read in the indexes. Negative filters are better than no filters at all; however these are only applied after the data is initially retrieved.

Make use of the CASE() and TERM() directives

By default, search filters are not case sensitive and will look for matches of data with specific segmentation. Segmentation within search filtering is typically split between two categories; major and minor breakers. Documentation for each can be found in Splunk Docs.

The CASE() directive can be used to increase the explicitness of a filter by ensuring it is case sensitive. The TERM() directive will only use major breakers which typically offers a big performance boost when searching as less resources will be consumed to identify any events. Instructions how to use each directive can be found in the Splunk Docs.

Regularly review and update filter configurations

As your data sources evolve and business needs shift, the criteria for what constitutes 'relevant data' may change. To ensure your filters remain aligned with organizational objectives:

  • Schedule periodic reviews of your filter configurations.
  • Consult with different teams (for example, security, operations, marketing) to understand their evolving data needs. This collaboration ensures filters serve broad organizational goals.

Avoid common pitfalls

Missteps in filtering can lead to loss of crucial data or waste of system resources. Here are common pitfalls to sidestep:

  • Over-Filtering: Especially at index-time, being too aggressive with filters can mean essential data never gets indexed, making recovery difficult.
  • Under-Filtering: Not filtering enough can strain system resources and clutter searches with irrelevant data.
  • Not Documenting Changes: Every change to your filtering criteria should be documented. This assists in troubleshooting and provides clarity for team members unfamiliar with prior configurations.
  • Relying Solely on Default Settings: Default settings in the Splunk platform are designed to be broadly applicable but may not be optimized for specific organizational contexts. Customize filter settings to match the unique data landscape of your enterprise.

Helpful resources

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at OnDemand-Inquires@splunk.com if you require assistance.

Limiting concurrent searches

While the Splunk platform is designed to handle vast amounts of data and deliver insights at speed, resources are not infinite. Without careful oversight, concurrent searches can rapidly consume available resources, leading to a noticeable slowdown in response times or even system instability. This section explains the significance of regulating concurrent searches and underscores how unmanaged searches can adversely impact the efficiency and overall performance of the Splunk platform.

►Click here to read more.

This section outlines the following steps in limiting concurrent searches:

  1. Understanding concurrent searches
  2. Assessing the risks of unrestricted concurrent searches
  3. Setting appropriate limits on concurrent searches
  4. Implementing best practices for efficient search management
  5. Monitoring and adjusting concurrent search limits

Understanding concurrent searches

The Splunk platform has powerful search capabilities that enable users to sift through vast datasets to extract meaningful insights. At the core of this capability is the concept of concurrent searches. But what exactly does this mean? Concurrent searches refer to multiple search queries executed simultaneously within your Splunk environment. Instead of processing searches sequentially, which would take a significant amount of time. The system is designed to handle several queries at once to maximize the utilization of resources.

To better understand how concurrent users and searches impact performance, see Scale your Splunk Enterprise deployment on Splunk Docs.

The characteristics of concurrent searches can vary based on the type of search (real-time vs historical), the complexity of the search, and the amount of data being queried. Some searches, due to their complexity or the sheer volume of data they encompass, might be resource-intensive, while others might be lightweight and quick.

For Splunk performance optimization, you need to prioritize the management of these concurrent searches. Just as a highway can only handle so many cars at once before traffic jams occur, the Splunk platform has a finite amount of resources to process searches. If too many heavy, resource-intensive searches run simultaneously, it can lead to bottlenecks. This can manifest as slower search return times, reduced system responsiveness, or even timeouts. That makes understanding and effectively managing concurrent searches not just a good-to-know concept, but a good skill for maintaining the health and efficiency of your Splunk environment.

Assessing the risks of unrestricted concurrent searches

Search capability in the Splunk platform is one of its most key features. However, when mismanaged, this strength can quickly become a liability. Allowing unrestricted concurrent searches can lead to a series of adverse effects on the system, users, and the overall reliability of the Splunk platform. Here are a few examples of issues that could be experienced:

  • Resource Depletion: Every search in the Splunk platform consumes system resources - CPU, memory, and I/O operations. When too many searches run simultaneously, especially if they are complex or span extensive datasets, there's a substantial risk of overwhelming the system. This can lead to slowed operations, lagging user interfaces, or even system crashes.
  • Degraded Search Performance: The Splunk platform is optimized to return results as quickly as possible. But if it's juggling too many concurrent searches, the processing power for each individual search is reduced. The result? Slower search times and, in extreme cases, search timeouts. For users, this can be frustrating, especially if timely insights are needed.
  • Increased Queue Times: The Splunk platform uses a queuing system to manage searches, especially during high-load periods. If unrestricted searches are allowed, the queue can become backlogged, meaning even simple searches could be delayed because they're waiting for resources to free up.
  • Reduced Data Integrity: With overwhelming search loads, there's a potential for incomplete or even incorrect data returns. For instance, if a search is terminated prematurely due to resource constraints, users might not get the full picture of their queried data.
  • Operational Inefficiencies: In enterprise settings, the Splunk platform often plays a pivotal role in decision-making processes. If searches are consistently slow, unreliable or completely absent due to unrestricted concurrency, it can introduce operational inefficiencies. Decisions might be delayed, or worse, made on incomplete data, even missed altogether.
  • Strain on Infrastructure: Continual maxing out of system resources can put a strain on the underlying infrastructure, leading to wear and tear on physical components or increased cloud costs, if hosted in a cloud environment.

While the Splunk platform has built-in mechanisms to manage concurrent searches, relying solely on defaults without understanding and fine-tuning based on the specific needs of an organization can be a risky approach. The goal should always be to strike a balance: ensuring users can get the data they need when they need it, without compromising the health and performance of the Splunk system.

Setting appropriate limits on concurrent searches

Ensuring optimal performance and preventing system overloads in the Splunk platform necessitates a cautious approach to managing concurrent searches. By setting appropriate limits, you not only maintain system responsiveness but also ensure that users can extract the data they need without undue delays. The needs of limits tend to be dynamic and can differ from one system to another. Further detail of the main assessment criteria and methods of application has been captured below.

Determining optimal concurrent searches for your environment

Before imposing any limits, take a some time to assess your environment. Consider the number of users, the complexity of typical searches, and the frequency of those searches. A small team running simple searches might require fewer concurrent search limits compared to a larger analytics-driven organization.

Factoring in system resources and user demands

Your system's CPU, memory, and I/O capabilities play a pivotal role in determining search limits. A more robust system might handle more concurrent searches than a resource-constrained one. Furthermore, gauge your user base: do they run intensive ad-hoc searches frequently, or are they more into scheduled, routine searches?

Example: If you have a quad-core CPU with 16GB RAM dedicated to the Splunk platform and typically notice only a fraction of this being used, you might consider allowing a higher number of concurrent searches. However, if you're frequently maxing out resources, it might be time to be stricter with your limits or consider a system upgrade.

Configuring search limits in the Splunk platform

The Splunk platform provides two primary mechanisms for setting search concurrency:

  1. Role-Based Search Concurrency: This allows you to set search limits based on user roles. For instance, you might allow administrators to run more concurrent searches than regular users, given their elevated requirements. To configure role-based concurrency:
    1. Edit the authorize.conf file located in $SPLUNK_HOME/etc/myapp/local/.
    2. Find the specific role.
    3. Adjust the value under the cumulativeSrchJobsQuota and srchJobsQuota section to desired setting.

    Example: For an analytics-heavy role, you might allow up to 10 concurrent searches, while a regular user role might be limited to three.

  2. System-Wide Search Concurrency: This sets a blanket limit across the entire Splunk environment, regardless of user role. To configure system-wide concurrency:
    1. Edit the limits.conf file located in $SPLUNK_HOME/etc/myapp/local/.
    2. Modify or add the [search] stanza with max_searches_per_cpu and base_max_searches attributes.

    Example:

    [search]
    max_searches_per_cpu = 4
    base_max_searches = 6

    This configuration allows four searches per CPU core with a minimum of six searches regardless of CPU core count.

For detailed steps and considerations, see Splunk official documentation on concurrent search limits.

Implementing best practices for efficient search management

Effectively managing searches in the Splunk platform ensures optimal system performance and achieving timely data retrieval. As your Splunk environment scales and user demands increase, instituting a set of best practices can make a significant difference in maintaining system efficiency.

Prioritizing critical searches

Identifying and prioritizing critical searches ensures that mission-critical operations have the resources they require. This can be accomplished by:

  • Setting Scheduled Search Priority: Use the scheduled search priority parameter in searches to prioritize them, with higher numbers indicating more importance.
  • Dedicated Search Heads: For extremely critical searches or dashboards, consider dedicating specific search heads. This ensures that vital operations are insulated from routine search activities.

Scheduling routine searches during off-peak times

Regular, routine searches can be resource-intensive, especially when executed frequently or across vast data sets. To minimize the impact on system performance:

  • Use Scheduled Searches: Schedule routine searches to run during times when the system faces lower user demand, like late nights or early mornings.
  • Leverage Report Acceleration: For recurring searches, enabling report acceleration can significantly speed up search execution by caching the results.

Educating users on efficient search practices and guidelines

End-users play a critical role in ensuring search efficiency. Providing them with guidance can lead to more effective search operations:

  • Training Sessions: Hold regular training sessions to educate users on efficient search syntax, use of wildcards, time frames, and other Splunk-specific functionalities.
  • Documentation: Create and distribute user-friendly documentation that provides tips, best practices, and examples of efficient searches.
  • Feedback Mechanism: Establish a feedback loop where users can report slow searches or seek optimization advice. This promotes continuous learning and optimization.

Monitoring and adjusting concurrent search limits

Efficiently managing performance of the Splunk platform involves not only setting concurrent search limits, but also regularly monitoring and adjusting them based on evolving system needs and business requirements. Effective monitoring helps in preempting potential issues, ensuring smooth system operations, and optimizing resource allocation. Here's a closer look at the facets of this process:

Tools and metrics for monitoring search performance and concurrency

The Splunk platform provides some tools and metrics that enable tracking of search performance and concurrency.

  • Monitoring Console: The Splunk Monitoring Console offers a consolidated view of the system's health and performance metrics, including those related to search concurrency.
    • Navigate to: Settings > Monitoring Console > Search > Activity > Instance. This dashboard provides a real-time overview of active, queued, and skipped searches, aiding in quick diagnosis and action.
  • Search Job Inspector: For detailed insights into individual search performance, use the Search Job Inspector. It reveals specifics about the search's execution, including time taken, scanned events, and used resources.
  • Metrics Log: The metrics log contains granular data about the Splunk platform operations. Queries targeting this log can discover insights into search concurrency, performance bottlenecks, and more.

Regularly reviewing and adjusting search concurrency settings

Static configurations rarely meet the dynamic demands of a growing Splunk environment. Regular reviews ensure that search concurrency settings remain aligned with the system's capabilities and your organization's needs.

  • Assess System Load: Regularly monitor the system's CPU, memory, and I/O performance. If the system is consistently under heavy load during peak search times, consider adjusting search limits.
  • Factor in New Business Requirements: As your organization grows and use cases evolve, search demands might change. Maybe there are more users, or perhaps certain new searches have become business-critical. Regularly reassess and realign search concurrency limits in light of these changes.
  • Engage with End-Users: Maintain open communication channels with Splunk end-users. Their feedback about search performance, delays, or system responsiveness can be pivotal in making informed adjustments.
  • Regularly Review Implemented Use-Cases: Just as there is a need to monitor the load to cater for new use cases. Reviewing business requirements and engaging with end-users is also necessary to ensure that any implemented use-cases are regularly reassessed and validated so as to not reduce resource efficiency.

Helpful resources

Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at OnDemand-Inquires@splunk.com if you require assistance.