Comparing search head performance gains after upgrading a low-utilization cluster

The hardware differences

The servers old-004 to old-006 used throughout this article are the same "type" of CPU (base clock 2.70 Ghz, maximum turbo of 3.70 Ghz). Servers new-004 to new-006 are the newer type of hardware (base clock 3.60 Ghz, maximum turbo of 4.10 Ghz).

Summary of findings

The following sections detail the differences between comparing scheduled searches versus ad-hoc search runs. This image shows the main findings for scheduled search runtimes—the average search head performance improvement over 7 days—broken down by runtime:

This image shows the average search head performance improvement over 30 days, broken down by runtime:

The improvement here is minimal. However, the graphs above do not isolate searches spending the majority of their time in phase 1, and the indexing tier is unchanged, so phase 0 should be roughly the same during the testing period. The difference is visible but much less than expected; a 2% average improvement seems quite low for the newer hardware combination.

If we change the ratio to less than 50% (so at least half of the overall search time is spent in phase 1), we see the following: the average search head performance improvement over 7 days, broken down by runtime, with longer phase 1 times:

This image shows the average search head performance improvement over 30 days, broken down by runtime, with longer phase 1 times:

Looking over 30 days was expected to show a larger improvement, but it simply decreased the percentage, and in some cases produced a negative result (which might be attributable to chance).

Dashboard performance changes were also graphed, but there was not enough data in this environment to draw any conclusions. In an alternative environment, dashboards did tend to see a larger benefit than saved searches (closer to 20% in many cases, while the saved searches saw less than 5% improvement).

Variance between scheduled search runtimes

One difference from the previous search head cluster and article is the consistency between scheduled search runs. This image shows the average phase 0 variance between runs over 7 days:

This image shows the average phase 0 variance between runs over 30 days:

This image shows the average phase 0 variance between runs over 7 days, broken down by runtime:

This image shows the average phase 0 variance between runs over 30 days, broken down by runtime:

This image shows the average phase 1 variance between runs over 7 days, broken down by runtime:

This image shows the average phase 1 variance between runs over 30 days, broken down by runtime:

This image shows the average phase 1 variance between runs over 7 days:

This image shows the average phase 1 variance between runs over 30 days:

The majority of searches on this cluster show a relatively close runtime between runs. In the previous test, there were results with around 30% variance between search runs, and quite a lot at 40% or more. In this environment, a lot of the searches were within 10% of their phase 1 runtime, with a little more variance from the indexing tier.

This image shows the average phase 0 variance by search type over 7 days:

This image shows the average phase 0 variance by search type over 30 days:

This image shows the average phase 1 variance by search type over 7 days:

This image shows the average phase 1 variance by search type over 30 days:

In terms of gathering this data, the search was extremely slow:

• 29,308 results by scanning 10,602,528 events in 1,256.504 seconds
• 46,350 results by scanning 46,026,829 events in 5,705.176 seconds

For the 30 day search, auto-finalization kicked in due to reaching the disk quota, and the search had to be re-run. The search used to determine these results clearly requires optimization, but it is only intended for occasional use.

Results from ad-hoc testing

For ad-hoc testing, the following queries were run:

index=_internal earliest=-1m
index=_internal earliest=-10m | stats count BY host, source
index=_internal earliest=-60m | stats count BY host, source

This image shows a graph of the percentage improvement per type of ad-hoc search (all in Smart mode):

The pure events search run in Smart mode returned a large volume of results and showed the largest improvement, close to a 50% decrease in runtime across the testing. This is likely helped by the faster I/O write performance. The stats-based queries showed less than 20% improvement, which is likely the CPU difference taking effect.

Updates to the testing method

The query has been updated to include dashboard performance this time, as dashboard performance appears to vary more (in this SHC) than the saved searches, and it is more noticeable to the userbase:

| multisearch 
    [ search index=_introspection sourcetype=search_telemetry `searchheadhosts` perf.search_runtime_secs>2 search_id!=1*
    | eval hour=strftime(_time, "%H"), minute=strftime(_time, "%M"), dayofweek=strftime(_time, "%a") 
    | rename desc.app AS app
    | fields phases.phase_0.elapsed_time_aggregations.max, phases.phase_1.elapsed_time_aggregations.max, host, search_id app ] 
    [ search index=_audit `searchheadhosts` info=completed total_run_time>2 search_id="'scheduler*" has_error_warn=false fully_completed_search=true 
    | eval hour=strftime(_time, "%H"), minute=strftime(_time, "%M"), dayofweek=strftime(_time, "%a") 
    | eval search_id=trim(search_id,"\'")
    | rex ", savedsearch_name=\"(?P<savedsearch_name>.*?)\", search_startup_time"
    | rex "(?s), search='(?P<search>.*)\]$" 
    | rex field=search mode=sed "s/\n/ /g" 
    | eval search=if(substr(search,len(search),len(search)-1)=="'",substr(search,0,len(search)-1),search)
    | rex field=search "^(\s*\|\s*)?\s*(?P<command>\S+)" 
    | rex field=search "(\||^)\s*(?P<subsearch>(append|union|multisearch|set|appendcols|appendpipe|join|map))"
    | eval subsearch=if(isnotnull(subsearch),"true","false")
    | eval search_type=case(command="ldapsearch" OR command="runaql" OR command="dbxquery","remote_server",command="tstats" OR command="metadata" OR command="eventcount","metadata",command="makeresults" OR command="inputlookup" OR command="rest" OR command="loadjob" OR command="noop" OR command="stats","local",match(command,"^trackme"),"customcommand",command="multisearch" OR command="union" OR command="savedsearch" OR command="datamodel" OR command="from" OR command="set","search",command="`","search",1=1,command)
    | eval search_source="savedsearch"
    | fields total_run_time, savedsearch_name, search_id, result_count, event_count, scan_count, search_type, app, subsearch, search_source ] 
    [ search index=_audit `searchheadhosts` info=completed total_run_time>2 has_error_warn=false fully_completed_search=true provenance="UI:Dashboard:*"
    | eval hour=strftime(_time, "%H"), minute=strftime(_time, "%M"), dayofweek=strftime(_time, "%a") 
    | eval search_id=trim(search_id,"\'")
    | eval time_period=round(api_lt-api_et) 
    | eval time_period = case(
        time_period >= 1296000, ceil(time_period/86400)*86400,
        time_period >= 14400 , ceil(time_period/60/60)*3600,
        time_period >= 300, ceil(time_period/60)*60,
        true(), time_period
        )     
    | eval time_period=if(api_et=="N/A","alltime",time_period)
    | rex ", savedsearch_name=\"(?P<savedsearch_name>.*?)\", search_startup_time"
    | eval savedsearch_name=provenance . "_" . savedsearch_name . "_" . time_period
    | rex "(?s), search='(?P<search>.*)\]$" 
    | rex field=search mode=sed "s/\n/ /g" 
    | eval search=if(substr(search,len(search),len(search)-1)=="'",substr(search,0,len(search)-1),search)
    | rex field=search "^(\s*\|\s*)?\s*(?P<command>\S+)" 
    | rex field=search "(\||^)\s*(?P<subsearch>(append|union|multisearch|set|appendcols|appendpipe|join|map))"
    | eval subsearch=if(isnotnull(subsearch),"true","false")
    | eval search_type=case(command="ldapsearch" OR command="runaql" OR command="dbxquery","remote_server",command="tstats" OR command="metadata" OR command="eventcount","metadata",command="makeresults" OR command="inputlookup" OR command="rest" OR command="loadjob" OR command="noop" OR command="stats","local",match(command,"^trackme"),"customcommand",command="multisearch" OR command="union" OR command="savedsearch" OR command="datamodel" OR command="from" OR command="set","search",command="`","search",1=1,command)
    | eval search_source="dashboard"
    | fields total_run_time, savedsearch_name, search_id, result_count, event_count, scan_count, search_type, app, subsearch, search_source ]
| eval type="new"
| selfjoin search_id app
| table _time, phases.phase_0.elapsed_time_aggregations.max, phases.phase_1.elapsed_time_aggregations.max, host, savedsearch_name, total_run_time, result_count, event_count, scan_count search_type app subsearch search_source
| bin _time span=24h 
| stats count, avg(phases.phase_1.elapsed_time_aggregations.max) AS avg_sh_time, avg(phases.phase_0.elapsed_time_aggregations.max) AS avg_indexer_time, avg(total_run_time) AS total_run_time, avg(scan_count) AS scan_count, avg(event_count) AS event_count, avg(result_count) AS result_count, values(subsearch) AS subsearch, stdev(phases.phase_1.elapsed_time_aggregations.max) AS phase_1_stdev, stdev(phases.phase_0.elapsed_time_aggregations.max) AS phase_0_stdev by savedsearch_name, type, _time, search_type, app, search_source
| eval phase_1_diff=round(phase_1_stdev/avg_sh_time,3), phase_0_diff=round(phase_0_stdev/avg_indexer_time,3)
| eventstats perc95(phase_1_diff) AS phase_1_diff, perc95(phase_0_perc) AS phase_0_diff, avg(phase_0_diff) AS avg_phase_0_diff, avg(phase_1_diff) AS avg_phase_1_diff by savedsearch_name, search_type, app
| eval day=strftime(_time, "%a") 
| eval hour=strftime(_time, "%H")
| sort 0 savedsearch_name type
| streamstats window=1 global=f current=f reset_on_change=true values(avg_sh_time) AS avg_new_sh_time, values(avg_indexer_time) AS avg_new_indexer_time, values(scan_count) AS prev_scan_count, values(event_count) AS prev_event_count, values(result_count) AS prev_result_count by savedsearch_name
| eval shc_diff=((avg_sh_time/avg_new_sh_time)-1)*100, indexer_diff=((avg_indexer_time/avg_new_indexer_time)-1)*100
| eval ratio=(100/avg_sh_time)*avg_indexer_time
| foreach avg* phase_* ratio shc_diff total_run_time [eval <<FIELD>>=round('<<FIELD>>',2)]
| rangemap field=avg_sh_time 1-3=1-3 4-6=4-6 7-10=7-10 11-30=11-30 31-60=31-60 61-120=61-120 121-300=121-300 301-600=301-600 default=over600

Conclusion

The expectations had already been adjusted after the previous test to expect a small difference. However, the combination of faster I/O and faster CPU was expected to make a more noticeable difference to performance.

While some searches did appear to have a 20% or greater benefit, the overall benefits in scheduled search performance were, on average, very small. The manual comparisons showed clear benefits for a simple query, but this does not translate to the saved searches.

Overall, the newer hardware does help improve performance, but the benefits for scheduled searches are minimal—even in this particular SHC, which has an extremely small user base of approximately 8 active users.

The previous article describes the testing methods and a more complicated comparison across other search head clusters.

Additional resources

These resources might help you understand and implement this guidance:

• Splunk Lantern Article: Comparing search head hardware changes and their impact on Splunk platform search performance
• Splunkbase: Metricator for Nmon
• Splunkbase: Alerts for Splunk Admins

• Splunk OnDemand Services: Use these credit-based services for direct access to Splunk technical consultants with a variety of technical services from a pre-defined catalog. Most customers have OnDemand Services per their Success Plan. Engage the ODS team at ondemand@cisco.com if you would like assistance.