Creating efficient searches and dashboards for cost reduction

Refactoring search queries

Optimizing search queries directly impacts TCO by reducing resource consumption and enhancing performance:

• Reduced Data Processing: By assigning specific time ranges through role permissions, you can control and limit the scope of data users can access and search. As a result, costs associated with processing and storage are reduced, and users are provided with a more focused and relevant dataset based on their roles.
• Positive Matching: Using positive matching to filter data reduces the amount of data retrieved from the indexers. While it is possible to filter data using negative arguments (for example, NOT), results are only filtered when data is collected from the indexers which results in inefficient searching. This can also affect search results.
• Efficient Resource Utilization: Filtering data early in the search pipeline minimizes the amount of data processed by subsequent search commands. This reduces the strain on system resources, leading to improved performance and lower infrastructure costs.
• Streamlined Data Analysis:
  • Aggregation commands such as stats and chart help consolidate and summarize data, reducing the volume of data processed. This optimization results in faster search execution, reducing the need for extensive computational resources and lowering infrastructure costs.
  • MapReduce allows for the efficient segregation of large volumes of data across various nodes (the 'map' phase), followed by the aggregation and synthesis of this data to derive meaningful insights (the 'reduce' phase). By distributing the computational load, MapReduce ensures that the Splunk platform can sift through logs and datasets effectively.

Leveraging search commands

Utilizing Splunk's search commands efficiently enhances search performance and contributes to TCO reduction:

• Optimized Field Manipulation: Leveraging eval commands for field calculations and formatting improves data analysis efficiency. By preparing data for analysis during the search phase, subsequent processing steps can be simplified, reducing overall resource consumption and lowering TCO.
• Streamlined Data Analysis: Aggregation commands such as stats and chart help consolidate and summarize data, reducing the volume of data processed. This optimization results in faster search execution, reducing the need for extensive computational resources and lowering infrastructure costs.

Optimizing search jobs

Efficient search job management minimizes resource waste and contributes to TCO reduction:

• Controlled Result Sizes: Limiting the number of search results using commands like head or top reduces memory consumption and improves search performance. By managing result sizes, organizations can optimize infrastructure costs while still obtaining the necessary insights from the data.
• Workload Management: Workload Management can be used to ensure the most important jobs are not resource constrained. For more information, see Workload management in the Reducing your infrastructure footprint pathway.
• Identifying Unnecessary Scheduled Saved Searches: Identifying and cleaning up unnecessary scheduled saved searches, especially out-of-the-box (OOTB) scheduled saved searches that aren't required, streamlines the system and trims infrastructure costs. This includes removing redundant or unused saved searches in applications like Splunk Enterprise Security (ES) or Splunk ITSI (ITSI).
• Resource Cleanup: Properly managing search job lifecycles, including canceling or terminating unnecessary or long-running jobs, prevents resource waste and optimizes system performance. This optimization reduces infrastructure costs by eliminating unnecessary resource usage.
  • Monitoring Console or the Cloud Monitoring Console can aid in this review process by providing at-a-glance insights into system health and performance. Regularly optimize, update, or remove searches based on changing needs.
  • Search Job Monitoring: Regularly monitor the ongoing search jobs in your Splunk environment. You can do this through the Splunk Search Job Inspector, which provides insights into active and historical search jobs.
  • Scheduled Searches Review: Examine your scheduled searches and reports. Determine whether all scheduled searches are still relevant and producing valuable insights. If there are reports that are rarely accessed or no longer provide significant value, consider discontinuing or optimizing them.
  • Stuck or Abandoned Jobs: Keep an eye out for search jobs that are stuck, running indefinitely, or have been abandoned. Canceling or terminating these jobs can free up resources.
  • Audit Search Usage: Review the usage and popularity of saved searches and reports. If certain searches are hardly ever used by users or teams, they might be candidates for optimization or removal.
  • Regular Review: Conduct periodic reviews of your search jobs to ensure they align with your organization's goals and requirements. Regularly optimize, update, or remove searches based on changing needs.

The optimizations discussed here result in lower infrastructure costs, reduced hardware requirements, improved operational efficiency, and increased productivity. By embracing these best practices, organizations can extract maximum value from their Splunk deployments while minimizing total cost of ownership, ultimately leading to a more cost-effective and efficient data analysis environment.

Optimizing dashboards

Well-optimized dashboards not only improve user experience but also contribute to TCO reduction:

• Resource Optimization: Consolidating multiple panels into a single panel reduces resource consumption and enhances dashboard loading times. This optimization translates to lower hardware requirements, reducing infrastructure costs.
• Base and Chain Searches: Base and chain searches encapsulate common search logic and filters, which can be reused across multiple searches and dashboards. This approach minimizes redundant code, improves consistency, and simplifies maintenance by executing a single search and reusing the returned data through other sub-searches. By leveraging base and chain searches, organizations reduce development time and effort, optimize resource utilization, and enhance performance.
• Accelerated Data Models: Utilizing data model acceleration or reducing the data set searched improves search and dashboard performance. Faster rendering and reduced computational demands result in lower resource consumption, leading to cost savings in terms of infrastructure and operational efficiency.
• Efficient Data Access: By leveraging summary indexing for frequently used searches, dashboards can load faster and require fewer computational resources. This optimization minimizes the need for extensive data retrieval, reducing infrastructure costs and improving overall dashboard performance.

Next steps

This article is part of the Splunk Outcome Path, Reducing your infrastructure footprint. Click into that path to find more ways you can maximize your investment in Splunk software and achieve cost savings.

In addition, these resources might help you implement the guidance provided in this article:

• Product Tip: Optimizing Search

• Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their license support plan. Engage the ODS team at ondemand@splunk.com if you require assistance.