Skip to main content
Splunk Lantern

Upgrading Splunk Enterprise

Splunk adds and updates features and functions regularly to keep pace with innovation and reduce risk. If you have Splunk Cloud Platform, these updates are released continually and you can read about recent changes in the Frequently Asked Questions section of this article. But with Splunk Enterprise on-prem, many companies have a policy to wait a few releases before upgrading any software to make sure that new features are stable. Waiting that long means the company then only has a year before that version moves out of support.

The software support policy for Splunk Enterprise is two years. For on-prem customers, Splunk releases two levels of software updates to Splunk Enterprise:

  • Major "x." and Minor "x.x" releases include new and updated features and functions, updated platform elements, and cumulative bug fixes. Splunk issues major and minor releases on average twice per year. 
  • Maintenance "x.x.x" releases include bug fixes and minor updates within a major release interval. Splunk issues maintenance releases several times per year.

On-prem customers benefit from the continual updates to Splunk Cloud Platform because features, functions, and updates are thoroughly road-tested and hardened when they are released in a major version update.

Considerations for upgrading

Version compatibility: What works with what? 

During upgrade planning, one of the first things to check is the compatibility of your Splunk premium apps and forwarders with your Splunk Enterprise destination version. Bookmark and use the Splunk versions compatibility matrix - a single place where you can look up which versions work together.

The following table shows compatibility between different forwarder and deployment server versions.

Forwarder version Splunk Enterprise deployment server version
7.x 9.x
8.x 9.x
9.x 9.x

Pitfalls: What can cause problems during an upgrade? 

You will find a list of all major pitfalls and considerations listed in About Upgrading READ THIS FIRST in the Splunk Enterprise Installation Manual, including:

  • Changes that can potentially break Splunk Enterprise installations
  • Occurrences that appear as problems but are normal
  • Considerations for changed or removed features
  • Considerations for new features

If you need help to support your organization through an upgrade, you can contact experts in Professional Services that can help you at any point. You can learn more about Splunk's technical advisory options through our Customer Success webpages.

Order of operations: What is the upgrade process? 

Learn about the process with the comprehensive General process to upgrade Splunk Enterprise diagram on Splunk Answers that gives a high-level overview of how to plan and execute a Splunk Enterprise upgrade. The graphic includes where apps and add-ons fit in, when to upgrade forwarders, and links to documentation topics that explain what to do. This general process works for all Splunk Validated Architectures - just skip the components that aren't relevant to your deployment.

When you are ready to begin, follow the steps laid out in Splunk Documentation for upgrading to the latest Splunk Enterprise version: How to Upgrade Splunk Enterprise.

You can check in with your Customer Success Manager or your Splunk account team to have them validate your upgrade plan and help you fine-tune it. Need help with planning this upgrade? Engage OnDemand Services directly to execute an "Upgrade Readiness Assessment" task or "Ask an Enterprise Expert" general consultative task to clarify questions you may have. Most customers have access to ODS per their license support contract. If you cannot open a request or do not know if you have OnDemand access, contact the ODS team at OnDemand-Inquires@splunk.com for assistance.

Best practices: What should you keep in mind during upgrade planning?

  • Upgrade your on-premise Splunk Enterprise implementation as often as you can, but at least once per year. Then establish a regular upgrade cadence so you can keep pace with all major and maintenance updates Splunk releases. If you are on a later Splunk Enterprise version, maintenance updates are straightforward and non-disruptive to perform. 
  • If Splunk issues a maintenance release during your upgrade planning cycle, we strongly recommend that you make the latest release the target of your upgrade.
  • Be proactive. Don't wait until you encounter a bug or reach the end of the version support window. 
  • If you have a clustered architecture, rolling upgrade features available since Splunk 7.1 have taken a lot of tedium out of regular upgrades. For details, see Perform a rolling upgrade of a search head cluster
  • Use the Upgrade Readiness App to prepare your Splunk platform deployment for upgrades to the latest version of Python, jQuery libraries etc. The Upgrade Readiness app is installed by default on Splunk Enterprise version 8.2 and higher. If you are on a Splunk Enterprise version prior to 8.2 but after 7.3, you can download the app from Splunkbase to install it on your deployment.

FAQ

These frequently asked questions address the most commonly asked questions from Splunk's June 2022 security advisories that can be addressed by upgrading to Splunk Enterprise 9.0. For our Splunk Cloud Platform customers, many of these fixes will be addressed by Splunk. See our Splunk Product Security page for the most up-to-date information and subscribe to get timely updates.

General questions for all customers

  1. Why should I upgrade to Splunk Enterprise 9.0 and/or update my Universal Forwarder?
    Splunk Enterprise 9.0 specifically includes new security features, a series of automatically implemented security settings, and addresses security vulnerabilities with fixes. 

    New or enhanced security features:
    • *New* Splunk Assist: a single place to monitor your Splunk Enterprise deployment and see recommendations to improve your security posture
    • *Enhanced* Upgrade Readiness App:  easily identify apps impacted by Python 3.0 certificate validation and access step by step upgrade guidance
    • *New* Smart Card Authentication: Splunk now supports multi-factor authentication (MFA) Common Access Card cryptographic certificates issued by the US DoD as a form of authentication natively

    Automatically implemented security updates settings:
    • Improved dashboard security with sanitization on input fields 
    • Increased control for admins with greater Splunk Enterprise roles and capabilities restriction options
    • Enhancements to third-party packages including Node.js, OpenSSL, and many more
    • Easier risk management with user-friendly SPL safeguards
    • Role-based field filtering

    Improved security posture:
    Customers are at the center of everything we do at Splunk and security is our top priority. On top of a series of new and improved capabilities, Splunk Enterprise 9.0 includes security upgrades & usability enhancements to address vulnerabilities and improve customers’ security posture, including:
    • Added hostname validation for Intra-Splunk Transport Layer Security (TLS): promoting the use of industry-standard encryption protecting data in transit across your Splunk architecture
    • Secure forwarders and deployment servers: enabling customers to apply stronger security at the forwarder as well as between the deployment server and forwarder
    • Added user-friendly search processing language (SPL) safeguards: protecting against data exfiltration and arbitrary code execution using SPL
    • App model hardening: enabling customers to apply stronger security between the application and Splunk system
     
    We strongly encourage customers to upgrade Splunk Enterprise and the Universal Forwarder to 9.0 as soon as possible to ensure the strongest security posture. For more details and guidance on next steps, please see our Product Security page, watch our Tech Talk - Improve Your Security Posture, view our documentation, and ask any additional questions at Splunk Answers on our Community site.
     
  2. What vulnerabilities were disclosed?
    See the Product Security page for more information.
     
  3. What products are affected by the vulnerabilities mentioned in the Security Advisories? 
    Splunk products that were affected by the identified vulnerabilities are listed in each Security Advisory. See Splunk Product Security for the list. The advisories released on June 14, 2022, impact Splunk Enterprise, Universal Forwarders, and the Splunk Cloud Platform. 
     
  4. Have the vulnerabilities been fully remediated? Are fixes available to customers?
    Splunk released patches for Splunk Enterprise on-prem and Universals Forwarders in the 9.0 release. For Splunk Cloud Platform, the fixed versions are listed in each advisory.
     
  5. Do I need to configure anything to remediate the issue?
    Most of the vulnerabilities require additional configurations to enable the remediation. Please refer to each advisory for the requisite call to action.
     
  6. How severe or impactful are the vulnerabilities?
    Each advisory lists the CVSSv3.1 vector, score, and severity. 
     
  7. Do the vulnerabilities impact Universal Forwarders?
    SVD-2022-0605 and SVD-2022-0606 impact Universal Forwarders directly. To remediate SVD-2022-0607 in your Deployment Server (DS) requires updating all UFs managed by those DSs to 9.0 before enabling the fix. 

    9.0 UF are compatible with 9.0, 8.1, and 8.2 Indexers for Splunk Enterprise on-prem and 8.1 and 8.2 Indexers for Splunk Cloud Platform. Also, 8.2.x/8.1.x UFs are compatible with 9.0 Deployment Servers.
     
  8. Does this impact heavy-weight forwarders (HWF)?
    Yes. The vulnerabilities impact HWFs. We recommend updating HWFs to the 9.0 release and enabling all additional changes.

  9. What can I do to detect the vulnerabilities?
    For Splunk Enterprise customers, Splunk provided detections through the Enterprise Security Content Updates (ESCU) Essential application to detect the vulnerability in your environment.
    For Splunk Cloud Platform customers, Splunk implemented the detections and continues to monitor for threats.
     
  10. Do the vulnerabilities impact older versions of Splunk or versions not listed?
    For Splunk Enterprise on-prem and Universal Forwarders, the vulnerabilities impact all versions before 9.0.
    For Splunk Cloud Platform, each advisory lists the fix version. All prior versions are impacted by the vulnerability.
     
  11. Are these vulnerabilities being actively exploited?
    We have no evidence of exploitation of the vulnerabilities by any external parties.
     
  12. Are there other vulnerabilities Splunk is aware of and has not disclosed? What is your disclosure policy? 
    Splunk follows industry best practices to discover and remediate vulnerabilities before disclosure. For Splunk’s disclosure policy, see Product Security at Splunk.
     
  13. Why is Splunk releasing the security advisories now?
    Splunk publishes out-of-band advisories for vulnerabilities that are time-sensitive. The criticality of the vulnerabilities warranted an immediate disclosure in coordination with the fix release. 

    For more information on the timing of vulnerability disclosures and security advisories, please visit the Spunk Product Security page.
     
  14. What procedures did Splunk conduct to evaluate the impact?
    Splunk executed our standard threat and vulnerability management procedure, which includes a comprehensive analysis for indications of potential compromise. 
     
  15. Has Splunk identified any indication of a security incident, compromise/breach related to these vulnerabilities? 
    We have no evidence of exploitation of the vulnerabilities by external parties.
     
  16. Has Splunk identified any customers that have been impacted by the vulnerabilities? 
    We have no evidence of exploitation of the vulnerabilities by any external parties.
     
  17. Did Splunk change the design or implement enhanced measures in its secure product development practice as a result of identifying these vulnerabilities? 
    No. Splunk did not change the design of its controls related to its secure product development, patch management, and deployment.
     
  18. Who discovered these vulnerabilities?
    All of the issues addressed were discovered internally through Splunk security testing or our internal bug bounty program, underscoring our commitment to proactive security assurance. 
     
  19. I noticed that you will be publishing advisories on a quarterly basis, is that new for Splunk?
    Earlier this year, Splunk implemented a quarterly cadence of Security Advisories to help Splunk administrators plan for patches and upgrades on a predictable schedule.

    Quarterly Security Patch Updates are collections of security fixes for Splunk products that are published after fixes are available. Security Patch Updates are available to our customers for all applicable and supported versions. Splunk will publish out-of-band advisories for vulnerabilities that are time-sensitive as soon as possible.  

    Quarterly patch updates are published on the first Tuesday of Splunk’s fiscal quarter.  The next three dates are:
    • August 2, 2022
    • November 1, 2022
    • February 7, 2023

    For more information, please visit the Spunk Product Security page.

  20. Where can I get more information?
    If you need additional assistance, please leverage your standard Splunk Customer Support channels, create a new support case, or work with your account team.

Splunk Enterprise deployments

  1. How can I tell in Splunk Enterprise if I have upgraded?
    For help with identifying your Splunk Enterprise version, please refer to the Splunk Docs Troubleshooting Manual. Additionally, you may need to enable additional fixes to remediate some vulnerabilities. Please refer to each advisory.
     
  2. Will Splunk release a patch for earlier Splunk Enterprise and Universal Forwarder supported versions? Do you plan to backport the security updates to Splunk 8.1.x or 8.2.x versions?
    Splunk does not plan to provide a complete patch or maintenance release for earlier support versions (Splunk Enterprise 8.2 and 8.1 and UFs 8.2 through 7.0). For additional information on the backporting rationale and process, please review the Security Advisories for Splunk 9.0 blog post.

    However, backporting to 8.1.x and 8.2.x for the single critical deployment server vulnerability (SVD-2022-0608) was completed on June 30. The details can be found on the following Splunk Docs pages:
    We strongly encourage customers to update to 9.0 as soon as possible. While a move to a 9.0 release will take time and effort, we believe this is the best way to address the issue. There is a mitigation for the most critical issue related to the deployment server that we recommend taking as you prepare to upgrade to 9.0. See the table in the following question.
  3. Why didn’t Splunk backport all the security fixes in 9.0? 
    Our decision not to backport all of the security vulnerabilities and instead issue a major release was to signal material, breaking changes to product behavior. The intent was to be consistent with our major, minor, patch release policy. It was also guided by a high probability of breaking longstanding product behavior and adversely impacting customer production instances. As an example, SVD-2022-0607 (deployment servers allow unauthenticated bundle access) requires both the Deployment Server (DS) and Universal Forwarders (UF) be version consistent and doesn't allow for a mixture of UF versions. Our testing demonstrates that mitigating SVD-2022-0601 (Splunk Enterprise disabled TLS validation using the CA certificate stores in Python 3 libraries by default) would result in breaking existing customer apps that use private certificates. Similarly, SVD-2022-0605 (UF management services allows remote login by default), would have impacted the Splunk CLI and could have broken customer integrations with tools such as Chef, Ansible or other scripted automation. SVD-2022-0603 (Splunk Enterprise lacked TLS hostname certificate validation) requires customers to have properly configured x509 certificates across all Splunk nodes, including valid SAN and CN values. These changes, intended to make Splunk more secure, nonetheless require thoughtful deployment planning beyond the expectations of a patch release.

    We understand that not all of our customers will be able to upgrade to the latest release immediately. To reduce the severity of these vulnerabilities during the process of upgrade, we have published partial mitigations (see question 6 below), as additional security controls to reduce security exposure in the interim, and will continue to update guidance as applicable. 

  4. Why is SVD-2022-0608 an exception to the decision to backport to 8.x.1 / 8.x.2?
    SVD-2022-0608, is the single critical vulnerability released with the June
    14th Security Advisory that cannot be mitigated without turning off the deployment server, and after considerable analysis we believe this backport has the least risk of introducing a regression. We’re in the process of releasing a backport for the versions under support (8.1.x / 8.2.x). We had initially ruled out a backport due to the risk of introducing regressions to Splunk instances that co-locate other services with the deployment server (such as License Manager, Deployer, etc.). We’ve performed additional work to confirm that a 9.0 deployment server is compatible with older versions of Universal Forwarders, License Managers, etc. Customers who have a Deployment Server co-located should review the compatibility matrix in the Considerations for Upgrading section of this document to ensure the backported fix will work in your environment. We appreciate community's feedback and aim to be more responsive.
     
  5. What is Splunk's vulnerability policy and its commitment to fixing and backporting vulnerabilities post 9.0?
    We remain committed to helping customers identify and remediate security issues quickly. For “Critical” or “High” vulnerabilities we plan to provide advisories and any available patches as close to real-time as possible. For vulnerabilities considered “Moderate” or “Low Risk”, we’re planning quarterly releases of any available patches so that Splunk administrators can plan for patches and upgrades on a predictable schedule. Please note we have updated our Security Advisory Policy to help clarify what Security Patch Updates are and the details about their availability.
     
  6. I can’t upgrade my on-prem Splunk Enterprise deployment right now. What are my mitigations?
    Depending on the configuration or implementation, some of the vulnerabilities may not impact the instance. In addition, Splunk offers detections for some vulnerabilities. Please refer to each advisory.

    However, in all cases, we strongly encourage customers to update to 9.0 as soon as possible. There are no full mitigations available aside from upgrading. Partial mitigations are provided in the following table.
    Advisory Info Partial Mitigation
    SVD-2022-0601 Splunk Enterprise disabled TLS validation in Python 3 libraries by default
    • Review your apps for outbound calls that do not use TLS.
    • Setup infra/reverse-proxy to monitor for outbound service calls.
    • Review network layer controls.
    • Splunk will provide a detection search that lets customers look for TLS clients that do not pass hostname or proper certificate base validation.
    SVD-2022-0602 Splunk Enterprise lacked TLS cert validation for Splunk-to-Splunk communication by default
    SVD-2022-0603 Splunk Enterprise lacked TLS hostname validation
    • Set infra/reverse proxy to monitor for outbound service calls.
    • Actively review network layer controls (for example, firewall logs) to ensure hosts used in TLS connections are trusted.
    SVD-2022-0604 Risky commands warnings in Splunk Enterprise dashboards
    SVD-2022-0605 Universal Forwarder management services allow remote login by default

    Do one of the following:

    • Set server.conf [httpServer] disableDefaultPort = true  OR 
    • Set server.conf [general] allowRemoteLogin = never  OR 
    • Set web.conf [settings] mgmtHostPort = localhost:8089
    SVD-2022-0606 Splunk Enterprise and Universal Forwarder CLI connections lacked TLS cert validation
    • Ensure Splunk CLI communications are configured properly with valid certificates.
    • Review for correct config of valid TLS certs on each Enterprise node.
    • Keep nodes inside a VPC/VPN or network isolation.
    SVD-2022-0607 Splunk Enterprise deployment servers allow unauthenticated Forwarder bundle downloads
    • Disable the deployment server temporarily until it’s upgraded to 9.0. 
    • Place the deployment server behind a firewall (be sure to allow Splunkweb and Splunkd management port).
    • Configure pass4SymmKey on the deployment server(s) to prevent unauthenticated clients (those without the right pass4SymmKey) from connecting.
    • Set up TLS on the deployment server(s) and configure clients to authenticate the server using transport layer security.
    SVD-2022-0608 Splunk Enterprise deployment servers allow client publishing of Forwarder bundles 
    • Disable the deployment server temporarily until you update to 9.0.
    • Place the deployment server behind a firewall and reduce the reachability of the DS to specific IPs or network segments where possible. 

Splunk Cloud Platform deployments

  1. Is Splunk Cloud Platform impacted by these vulnerabilities? 
    Yes, SVD-2022-0601, SVD-2022-0602, and SVD-2022-0603 impact all Splunk Cloud Platform instances as of Jun 14, 2022. SVD-2022-0604 impacts a minority of Splunk Cloud Platform instances on 8.2.2106 and earlier.
     
  2. When will Splunk update my Splunk Cloud Platform instance and enable the fixes?
    Due to the complexity and potential impact of fully remediating a cloud deployment, roll out requires careful planning and coordination as to not disrupt customers. We aim to patch all customer cloud deployments aligned to our 4-6 weekly iterative release process by early September. 
     
  3. How can I tell in Splunk Cloud Platform if I have been upgraded?
    For help identifying your Splunk Cloud Platform versions, please refer to the Splunk Docs Troubleshooting Manual. However, even if your deployment is on the Emerald release, Splunk must perform additional steps to configure and enable the changes. 
     
  4. How will a Splunk Cloud Platform user know when configurations have been implemented? 
    These vulnerabilities will be patched during the upgrade process to Splunk Cloud Platform version 9.0.2205. If you are on a lower version of Splunk Cloud Platform, you must first be upgraded to 9.0.2205. Note that this upgrade does require a maintenance window. If you wish to verify the state of the feature flags you can run the following SPL query in your Splunk Cloud Platform environment.

    Splunk Cloud Platform Configurations SPL

    The following SPL allows a Splunk Cloud Platform customer to determine the enforce TLS configurations after updating to 8.2.2203. 

    | rest /services/configs/conf-server/kvstore  ```Returns the configuration of KVStore TLS Enforcement & if the config is set in /system or an app```
    | eval sslVerifyServerCert= if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert),  kvstore_configuredSystem=if(app_dir="system","true","false")  
    | stats values(sslVerifyServerCert) as kvstore_sslVerifyServerCert values(eai:acl.app) as kvstore_configuredApp by splunk_server 
    | eval kvstore_configuredSystem=if(kvstore_configuredApp="system","true",kvstore_configuredApp) 
    | fields kvstore_sslVerifyServerCert, splunk_server, kvstore_configuredSystem
    | append [
      | rest /services/configs/conf-server/sslConfig ```Returns the current configuration for enforcement of TLS Hostname Validation & if the config is set in /system or an app```
      | eval sslVerifyServerName=if(isnull(sslVerifyServerName),"unset",sslVerifyServerName) | stats values(sslVerifyServerName) as servername_sslVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server 
      | eval servername_configuredSystem=if(servername_configuredApp="system","true",servername_configuredApp)     
      | fields servername_sslVerifyServerName, splunk_server, servername_configuredSystem
    ]
    | append [
      | rest /services/configs/conf-server/sslConfig ```Returns the current configuration for enforcement of TLS Certificate validation for Splunk-to-Splunk communications & if the config is set in /system or an app```
      | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert) 
      | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server 
      | eval global_configuredSystem=if(global_configuredApp="system","true","false") 
      | fields global_sslVerifyServerCert, splunk_server, global_configuredApp 
    ]
    | append [ 
      | rest /services/configs/conf-server/pythonSslClientConfig ```Returns the current configuration for enforcement of Python TLS Certificate Validation & if the config is set in /system or an app - note that this can also be set in splunk-launch.conf but we cannot use search to read that config```
      | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert)
      | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert values(sslVerifyServerName) as python_sslVerifyServerName by splunk_server 
      | eval python_configuredSystem=if(python_configuredApp="system","true","false")
      | fields python_sslVerifyServerCert, python_sslVerifyServerName, splunk_server, python_configuredSystem
    ]
    | stats values(*) as * by splunk_server

    The output will look similar to the following:

    Note that values "unset" and "0" mean that you are still in "warning mode" for that particular feature flag:

    • "global_sslVerifyServerCert" for cert validation
    • "kvstore_sslVerifyServerCert" for KVStore cert validation
    • "servername_sslVerifyServerName" for TLS Hostname validation

    For Python TLS, there is no way to query this from the search bar completely, but the "python_sslVerifyServerCert" & "python_sslVerifyServerName" indicates the values for server.conf the stanza [pythonSslClientConfig]. However, you can also set this in splunk-launch.conf as well, which is not searchable via the | rest command in SPL.

Next steps