Reducing search load

Index-time filters

• What are index-time filters? Index-time filters are applied as data is being ingested into the Splunk platform. This means that data is filtered before it's even indexed, so only the data that meets the specific criteria makes it into the Splunk index. This data can either be routed to another index or sent to the nullqueue.
• When and why to use them: Index-time filters are especially useful when there's a need to exclude specific data from being stored in the Splunk platform to begin with, either for privacy reasons, regulatory compliance, or simply to optimize storage. For instance, you might exclude verbose logs from a particular system or sensor readings from a malfunctioning device, ensuring they don't consume valuable storage space or clutter up search results.
• Implement index-time filters: Before diving into the technical configurations, first decide which data to target with index-time filters. It's a strategic step: the goal is to identify data that, while perhaps available for ingestion, does not provide value when analyzed or stored. This might be because of redundancy, irrelevance, or even compliance mandates. Consider the following:
  • Are there log entries from a particular system or application that are excessively verbose and not useful for analysis?
  • Is there sensitive information that should never be stored in the Splunk platform for compliance or security reasons?
  • Are there event types or log sources that are irrelevant to the objectives of your Splunk deployment?
• Configuring index-time filters: After you have a clear idea of the data you'd like to filter out at index-time, the next step involves configuration. This is done using props.conf and transforms.conf configuration files. In props.conf, you'll specify the data source and direct it to a transformation in transforms.conf, which then performs the actual filtering. Here is an example process to follow:
  1. In props.conf, identify the data source using a stanza, then associate it with a transformation.
  2. In transforms.conf, define the transformation, specifying the filtering criteria.
  3. Ensure you carefully test any index-time filter configurations in a staging environment first, as these filters can't be undone without reindexing the data.
• Common examples and use cases
  • Redundant Data Removal: If you have a system that logs both a detailed and a summary event for every occurrence, but only the summary is relevant, an index-time filter can exclude the detailed logs.
  • Compliance-based Filtering: For regulatory reasons, you might want to exclude logs that contain sensitive personal information.
  • Infrastructure Noise Reduction: If certain machines or applications are known to produce "noisy" logs that don't contribute to meaningful analysis, these can be filtered at index-time.

By configuring index-time filtering, Splunk users can maintain a cleaner, more efficient, and more relevant dataset, ensuring that the system is not bogged down by unnecessary data and that storage costs are optimized. For more information on this topic, refer to Route and filter data on Splunk Docs.

Search-time filters

• What are search-time filters? Search-time filters are applied when querying already indexed data. Instead of pre-filtering data during ingestion, this filtering type sifts through existing indexed data, selecting only the subsets of data that align with the defined criteria for the search.
• Their advantages and use cases: Search-time filters provide the flexibility to dynamically refine search results without altering the underlying indexed data. They're invaluable when dealing with broad datasets where different queries might require different data subsets. For instance, an administrator could focus on logs from a particular server during a specific hour, or a security analyst might narrow down logs to specific IP addresses when investigating potential threats. Filtering types in the Splunk platform are tailored to address varied requirements, ranging from storage optimization with index-time filters to dynamic query refinement using search-time filters. Becoming better using these filters is a key competency for any Splunk user aiming for efficient and targeted data analytics.
• Implement search-time filters: Search-time filtering operates on the principle of refining the data you've already indexed when conducting a search, instead of during the ingestion process. As such, the first step is to determine the subsets of data that are pertinent to your current search objectives. It's worth noting that the choice to apply search-time filters can be situational, often guided by the specific task at hand. Consider the following:
  • What specific data points or events are you looking to analyze?
  • Are there indexed datasets that, while potentially useful in other scenarios, are extraneous to your current search?
• Configuring search-time filters: After identifying the desired data subsets, you'll transition to the Splunk search or the SPL (Splunk Processing Language) to implement the search-time filters. Using SPL, you can leverage various commands and functions, like search, where, and eval, among others, to narrow down the search results. For example, to filter events from a specific source or source type, you'd use a query like: source="your_source". Always ensure that your filter criteria are both precise and optimized for efficiency, as vague or overly broad search criteria can consume unnecessary system resources.
• Common examples and use cases
  • Security Monitoring: If you're monitoring for security events but want to exclude routine login and logout events, a search-time filter like NOT (eventtype IN ("login","logout")) could be applied.
  • Operational Metrics: When tracking the uptime of a server, irrelevant events like user logins or software updates could be filtered out, focusing solely on start-up and shutdown events.
  • Debugging: If you're diagnosing a system error, you might apply search-time filters to exclude all events except those flagged as 'error' or 'critical'.

Implementing search-time filters effectively is a valuable skill for Splunk users. It enables them to create tailored searches that address specific analytical requirements without sifting through mountains of unrelated data.

If you looking for help writing better searches in the Splunk platform and applying the concepts of this article, Writing better queries in Splunk Search Processing Language has a lot of great, actionable tips.

Time is the most important search filter

Data stored in indexes is typically stored in buckets which are time bound. The most efficient filter that can be applied to searches is that of time as it will reduce the number of events that need to be read across the indexes and eventually processed. For data stored in SmartStore indexes, this is even more important as older buckets are stored in the cloud. These buckets are automatically copied to a local cache whenever they are required for search purposes.

Ensure efficient filtering without losing critical data

When crafting filters, specificity is paramount. While it's tempting to craft broad filters for convenience, doing so might inadvertently screen out crucial data points. Always verify the impact of your filters, especially after initial implementation.

Specify what you want, rather than what you don’t want

Using positive filters to specify what you want will increase the efficiency of your searches because the system will be able to apply those filters directly to the data that needs to be read in the indexes. Negative filters are better than no filters at all; however these are only applied after the data is initially retrieved.

Make use of the CASE() and TERM() directives

By default, search filters are not case sensitive and will look for matches of data with specific segmentation. Segmentation within search filtering is typically split between two categories; major and minor breakers. Documentation for each can be found in Splunk Docs.

The CASE() directive can be used to increase the explicitness of a filter by ensuring it is case sensitive. The TERM() directive will only use major breakers which typically offers a big performance boost when searching as less resources will be consumed to identify any events. Instructions how to use each directive can be found in the Splunk Docs.

Regularly review and update filter configurations

As your data sources evolve and business needs shift, the criteria for what constitutes 'relevant data' may change. To ensure your filters remain aligned with organizational objectives:

• Schedule periodic reviews of your filter configurations.
• Consult with different teams (for example, security, operations, marketing) to understand their evolving data needs. This collaboration ensures filters serve broad organizational goals.

Avoid common pitfalls

Missteps in filtering can lead to loss of crucial data or waste of system resources. Here are common pitfalls to sidestep:

• Over-Filtering: Especially at index-time, being too aggressive with filters can mean essential data never gets indexed, making recovery difficult.
• Under-Filtering: Not filtering enough can strain system resources and clutter searches with irrelevant data.
• Not Documenting Changes: Every change to your filtering criteria should be documented. This assists in troubleshooting and provides clarity for team members unfamiliar with prior configurations.
• Relying Solely on Default Settings: Default settings in the Splunk platform are designed to be broadly applicable but may not be optimized for specific organizational contexts. Customize filter settings to match the unique data landscape of your enterprise.

Determining optimal concurrent searches for your environment

Before imposing any limits, take a some time to assess your environment. Consider the number of users, the complexity of typical searches, and the frequency of those searches. A small team running simple searches might require fewer concurrent search limits compared to a larger analytics-driven organization.

Factoring in system resources and user demands

Your system's CPU, memory, and I/O capabilities play a pivotal role in determining search limits. A more robust system might handle more concurrent searches than a resource-constrained one. Furthermore, gauge your user base: do they run intensive ad-hoc searches frequently, or are they more into scheduled, routine searches?

Example: If you have a quad-core CPU with 16GB RAM dedicated to the Splunk platform and typically notice only a fraction of this being used, you might consider allowing a higher number of concurrent searches. However, if you're frequently maxing out resources, it might be time to be stricter with your limits or consider a system upgrade.

Configuring search limits in the Splunk platform

The Splunk platform provides two primary mechanisms for setting search concurrency:

1. Role-Based Search Concurrency: This allows you to set search limits based on user roles. For instance, you might allow administrators to run more concurrent searches than regular users, given their elevated requirements. To configure role-based concurrency:
   1. Edit the authorize.conf file located in $SPLUNK_HOME/etc/myapp/local/.
   2. Find the specific role.
   3. Adjust the value under the cumulativeSrchJobsQuota and srchJobsQuota section to desired setting.

   Example: For an analytics-heavy role, you might allow up to 10 concurrent searches, while a regular user role might be limited to three.

2. System-Wide Search Concurrency: This sets a blanket limit across the entire Splunk environment, regardless of user role. To configure system-wide concurrency:
   1. Edit the limits.conf file located in $SPLUNK_HOME/etc/myapp/local/.
   2. Modify or add the [search] stanza with max_searches_per_cpu and base_max_searches attributes.

   Example:

   [search]
   max_searches_per_cpu = 4
   base_max_searches = 6

   This configuration allows four searches per CPU core with a minimum of six searches regardless of CPU core count.

For detailed steps and considerations, see Splunk official documentation on concurrent search limits.

Prioritizing critical searches

Identifying and prioritizing critical searches ensures that mission-critical operations have the resources they require. This can be accomplished by:

• Setting Scheduled Search Priority: Use the scheduled search priority parameter in searches to prioritize them, with higher numbers indicating more importance.
• Dedicated Search Heads: For extremely critical searches or dashboards, consider dedicating specific search heads. This ensures that vital operations are insulated from routine search activities.

Scheduling routine searches during off-peak times

Regular, routine searches can be resource-intensive, especially when executed frequently or across vast data sets. To minimize the impact on system performance:

• Use Scheduled Searches: Schedule routine searches to run during times when the system faces lower user demand, like late nights or early mornings.
• Leverage Report Acceleration: For recurring searches, enabling report acceleration can significantly speed up search execution by caching the results.

Educating users on efficient search practices and guidelines

End-users play a critical role in ensuring search efficiency. Providing them with guidance can lead to more effective search operations:

• Training Sessions: Hold regular training sessions to educate users on efficient search syntax, use of wildcards, time frames, and other Splunk-specific functionalities.
• Documentation: Create and distribute user-friendly documentation that provides tips, best practices, and examples of efficient searches.
• Feedback Mechanism: Establish a feedback loop where users can report slow searches or seek optimization advice. This promotes continuous learning and optimization.

Tools and metrics for monitoring search performance and concurrency

The Splunk platform provides some tools and metrics that enable tracking of search performance and concurrency.

• Monitoring Console: The Splunk Monitoring Console offers a consolidated view of the system's health and performance metrics, including those related to search concurrency.
  • Navigate to: Settings > Monitoring Console > Search > Activity > Instance. This dashboard provides a real-time overview of active, queued, and skipped searches, aiding in quick diagnosis and action.
• Search Job Inspector: For detailed insights into individual search performance, use the Search Job Inspector. It reveals specifics about the search's execution, including time taken, scanned events, and used resources.
• Metrics Log: The metrics log contains granular data about the Splunk platform operations. Queries targeting this log can discover insights into search concurrency, performance bottlenecks, and more.

Regularly reviewing and adjusting search concurrency settings

Static configurations rarely meet the dynamic demands of a growing Splunk environment. Regular reviews ensure that search concurrency settings remain aligned with the system's capabilities and your organization's needs.

• Assess System Load: Regularly monitor the system's CPU, memory, and I/O performance. If the system is consistently under heavy load during peak search times, consider adjusting search limits.
• Factor in New Business Requirements: As your organization grows and use cases evolve, search demands might change. Maybe there are more users, or perhaps certain new searches have become business-critical. Regularly reassess and realign search concurrency limits in light of these changes.
• Engage with End-Users: Maintain open communication channels with Splunk end-users. Their feedback about search performance, delays, or system responsiveness can be pivotal in making informed adjustments.
• Regularly Review Implemented Use-Cases: Just as there is a need to monitor the load to cater for new use cases. Reviewing business requirements and engaging with end-users is also necessary to ensure that any implemented use-cases are regularly reassessed and validated so as to not reduce resource efficiency.