Splunk 9.0.0 FAQ

General questions for all customers

1. Why should I upgrade to Splunk Enterprise 9.0 and/or update my universal forwarder?
   Splunk Enterprise 9.0 specifically includes new security features, a series of automatically implemented security settings, and addresses security vulnerabilities with fixes. 
   
   New or enhanced security features:
   • *New* Splunk Assist: a single place to monitor your Splunk Enterprise deployment and see recommendations to improve your security posture
   • *Enhanced* Upgrade Readiness App: easily identify apps impacted by Python 3.0 certificate validation and access step by step upgrade guidance
   
   Automatically implemented security updates settings:
   • Improved dashboard security with sanitization on input fields 
   • Increased control for admins with greater Splunk Enterprise roles and capabilities restriction options
   • Enhancements to third-party packages including Node.js, OpenSSL, and many more
   • Easier risk management with user-friendly SPL safeguards
   • Role-based field filtering
   
   Improved security posture:
   Customers are at the center of everything we do at Splunk and security is our top priority. On top of a series of new and improved capabilities, Splunk Enterprise 9.0 includes security upgrades & usability enhancements to address vulnerabilities and improve customers’ security posture, including:
   • Added hostname validation for Intra-Splunk Transport Layer Security (TLS): promoting the use of industry-standard encryption protecting data in transit across your Splunk architecture
   • Secure forwarders and deployment servers: enabling customers to apply stronger security at the forwarder as well as between the deployment server and forwarder
   • Added user-friendly search processing language (SPL) safeguards: protecting against data exfiltration and arbitrary code execution using SPL
   • App model hardening: enabling customers to apply stronger security between the application and Splunk system
    
   We strongly encourage customers to upgrade Splunk Enterprise and the universal forwarder to 9.0 as soon as possible to ensure the strongest security posture. For more details and guidance on next steps, please see our Product Security page, watch our Tech Talk - Improve Your Security Posture, view our documentation, and ask any additional questions at Splunk Answers on our Community site.
    
2. What vulnerabilities were disclosed?
   See the Product Security page for more information.
    
3. What products are affected by the vulnerabilities mentioned in the Security Advisories? 
   Splunk products that were affected by the identified vulnerabilities are listed in each Security Advisory. See Splunk Product Security for the list. The advisories released on June 14, 2022, impact Splunk Enterprise, universal forwarders, and the Splunk Cloud Platform. 
    
4. Have the vulnerabilities been fully remediated? Are fixes available to customers?
   Splunk released patches for Splunk Enterprise on-prem and universals forwarders in the 9.0 release. For Splunk Cloud Platform, the fixed versions are listed in each advisory.
    
5. Do I need to configure anything to remediate the issue?
   Most of the vulnerabilities require additional configurations to enable the remediation. Please refer to each advisory for the requisite call to action.
    
6. How severe or impactful are the vulnerabilities?
   Each advisory lists the CVSSv3.1 vector, score, and severity. 
    
7. Do the vulnerabilities impact universal forwarders?
   SVD-2022-0605 and SVD-2022-0606 impact universal forwarders directly. To remediate SVD-2022-0607 in your Deployment Server (DS) requires updating all UFs managed by those DSs to 9.0 before enabling the fix. 
   
   9.0 UF are compatible with 9.0, 8.1, and 8.2 Indexers for Splunk Enterprise on-prem and 8.1 and 8.2 Indexers for Splunk Cloud Platform. Also, 8.2.x/8.1.x UFs are compatible with 9.0 Deployment Servers.
    
8. Does this impact heavy forwarders?
   Yes. The vulnerabilities impact heavy forwarders. We recommend updating heavy forwarders to the 9.0 release and enabling all additional changes.


9. What can I do to detect the vulnerabilities?
   For Splunk Enterprise customers, Splunk provided detections through the Enterprise Security Content Updates (ESCU) Essential application to detect the vulnerability in your environment.
   For Splunk Cloud Platform customers, Splunk implemented the detections and continues to monitor for threats.
    
10. Do the vulnerabilities impact older versions of Splunk or versions not listed?
    For Splunk Enterprise on-prem and universal forwarders, the vulnerabilities impact all versions before 9.0.
    For Splunk Cloud Platform, each advisory lists the fix version. All prior versions are impacted by the vulnerability.
     
11. Are these vulnerabilities being actively exploited?
    We have no evidence of exploitation of the vulnerabilities by any external parties.
     
12. Are there other vulnerabilities Splunk is aware of and has not disclosed? What is your disclosure policy? 
    Splunk follows industry best practices to discover and remediate vulnerabilities before disclosure. For Splunk’s disclosure policy, see Product Security at Splunk.
     
13. Why is Splunk releasing the security advisories now?
    Splunk publishes out-of-band advisories for vulnerabilities that are time-sensitive. The criticality of the vulnerabilities warranted an immediate disclosure in coordination with the fix release. 
    
    For more information on the timing of vulnerability disclosures and security advisories, please visit the Spunk Product Security page.
     
14. What procedures did Splunk conduct to evaluate the impact?
    Splunk executed our standard threat and vulnerability management procedure, which includes a comprehensive analysis for indications of potential compromise. 
     
15. Has Splunk identified any indication of a security incident, compromise/breach related to these vulnerabilities? 
    We have no evidence of exploitation of the vulnerabilities by external parties.
     
16. Has Splunk identified any customers that have been impacted by the vulnerabilities? 
    We have no evidence of exploitation of the vulnerabilities by any external parties.
     
17. Did Splunk change the design or implement enhanced measures in its secure product development practice as a result of identifying these vulnerabilities? 
    No. Splunk did not change the design of its controls related to its secure product development, patch management, and deployment.
     
18. Who discovered these vulnerabilities?
    All of the issues addressed were discovered internally through Splunk security testing or our internal bug bounty program, underscoring our commitment to proactive security assurance. 
     
19. I noticed that you will be publishing advisories on a quarterly basis, is that new for Splunk?
    Earlier this year, Splunk implemented a quarterly cadence of Security Advisories to help Splunk administrators plan for patches and upgrades on a predictable schedule.
    
    Quarterly Security Patch Updates are collections of security fixes for Splunk products that are published after fixes are available. Security Patch Updates are available to our customers for all applicable and supported versions. Splunk will publish out-of-band advisories for vulnerabilities that are time-sensitive as soon as possible.  
    
    Quarterly patch updates are published on the first Tuesday of Splunk’s fiscal quarter.  The next three dates are:
    • August 2, 2022
    • November 1, 2022
    • February 7, 2023

    For more information, please visit the Spunk Product Security page.

20. Where can I get more information?
    If you need additional assistance, please leverage your standard Splunk Customer Support channels, create a new support case, or work with your account team.

Splunk Enterprise deployments

1. Is JQquery 3.5 a requirement for all apps in Splunk version 9.0?
   JQuery is completely separate from the 9.0 disclosures. However, our advice is to ensure that your apps are compatible with JQuery 3.5 and to ensure your dashboards follow the guidance outlined in the JQuery 3.5 public docs.
    
2. Regarding SVD-2022-0606, the partial mitigations provided mention configuring TLS hostname validation for Splunk CLI, which is only available in 9.0. Is there more information on how to configure valid certs in the doc? What does it mean by valid certs (e.g., self signed, internal CA, etc.)?
   Hostname validation for mitigation of SVD-2022-0606 varies depending upon the mode of use:
   • (Most common) If the Splunk CLI tool is only used for Splunk administrative commands on management nodes (e.g., splunk apply cluster-bundle, splunk reload deploy-server, etc.), then the certificate is checked for an IP address mapping of 127.0.0.1 (loopback). This can be configured with the X509 v3_req extensions supporting Subject Alternate Names within the certificate.
   • (Rarely) If the Splunk CLI is being used with the -uri parameter in order to manage a remote Splunk instance, then the calling host (where the CLI command is run) must be able to validate the remote instance’s certificate. Most commonly, this means that the local host must have a copy of the CA certificate with which the remote host’s certificate was signed.
      
3. What is the quickest way to address the critical 9.0 DS security vulnerability (SVD-2022-0608)?
   The fastest way to address this is to upgrade your deployment server to Splunk 9.0. The patch for this vulnerability only has also been backported to 8.1.10.1 and 8.2.6.1. Note that a deployment server on 9.0 is still compatible with lower versions of the universal forwarder. Additionally, if you have both deployment server and license manager running on the same node, Splunk supports a combined LM + DS node on 9.0 with the rest of the environment on lower versions (down to 8.1.x). Please note, however, that the feature flags associated with SVD-2022-0607 require that the universal forwarder and deployment server be on 9.0.
    
4. For SVD-2022-0608, if only the deployment server needs to be updated, is it possible/advisable to run a 9.0 deployment server with all other components being 8.2.x? What compatibility issues might arise?
   Splunk supports running a 9.0 deployment server with 8.2.x universal forwarders, as long as you do not need UF + DS authentication that is part of SVD-2022-0607. Per SVD-2022-0607, to support authentication between DS and UF, they both must be on 9.0 minimum.
    
5. Is it safe to upgrade to 9.0 if the deployment server is also my license master?
   Yes, you can upgrade the DS+LM node to 9.0 and still leave the other nodes (search head, indexer, etc.) on a lower version.
    
6. Regarding CVE-2022-32158 and CVE-2022-32157, I currently use the deployment server to control the heavy forwarders in my environment. I currently have no universal forwarders reporting to a deployment server. How are heavy forwarders connected to the deployment server affected by these CVEs? Should they be treated the same as the universal forwarder?
   The vulnerabilities listed here apply to any deployment client node, whether universal forwarder, heavy forwarder, license manager, search head, indexer, cluster master, search head cluster deployer, etc.
    
7. If my search head cluster deployer is on the same host as my deployment server, and my deployment server has been upgraded to 9.0, how can I apply shcluster-bundle?
   Add the -force true option to the apply command. Note that this may take longer than usual.
    
8. How can I tell in Splunk Enterprise if I have upgraded?
   For help with identifying your Splunk Enterprise version, please refer to the Splunk Docs Troubleshooting Manual. Additionally, you may need to enable additional fixes to remediate some vulnerabilities. Please refer to each advisory.
    
9. Will Splunk release a patch for earlier Splunk Enterprise and universal forwarder supported versions? Do you plan to backport the security updates to Splunk 8.1.x or 8.2.x versions?
   Splunk does not plan to provide a complete patch or maintenance release for earlier support versions (Splunk Enterprise 8.2 and 8.1 and UFs 8.2 through 7.0). For additional information on the backporting rationale and process, please review the Security Advisories for Splunk 9.0 blog post.
   
   However, backporting to 8.1.x and 8.2.x for the single critical deployment server vulnerability (SVD-2022-0608) was completed on June 30. The details can be found on the following Splunk Docs pages:
   • 8.1.10.1
   • 8.2.6.1
   
   We strongly encourage customers to update to 9.0 as soon as possible. While a move to a 9.0 release will take time and effort, we believe this is the best way to address the issue. There is a mitigation for the most critical issue related to the deployment server that we recommend taking as you prepare to upgrade to 9.0. See the table in the following question.

10. Why didn’t Splunk backport all the security fixes in 9.0? 
    Our decision not to backport all of the security vulnerabilities and instead issue a major release was to signal material, breaking changes to product behavior. The intent was to be consistent with our major, minor, patch release policy. It was also guided by a high probability of breaking longstanding product behavior and adversely impacting customer production instances. As an example, SVD-2022-0607 (deployment servers allow unauthenticated bundle access) requires both the Deployment Server (DS) and universal forwarders (UF) be version consistent and doesn't allow for a mixture of UF versions. Our testing demonstrates that mitigating SVD-2022-0601 (Splunk Enterprise disabled TLS validation using the CA certificate stores in Python 3 libraries by default) would result in breaking existing customer apps that use private certificates. Similarly, SVD-2022-0605 (UF management services allows remote login by default), would have impacted the Splunk CLI and could have broken customer integrations with tools such as Chef, Ansible or other scripted automation. SVD-2022-0603 (Splunk Enterprise lacked TLS hostname certificate validation) requires customers to have properly configured x509 certificates across all Splunk nodes, including valid SAN and CN values. These changes, intended to make Splunk more secure, nonetheless require thoughtful deployment planning beyond the expectations of a patch release.
    
    We understand that not all of our customers will be able to upgrade to the latest release immediately. To reduce the severity of these vulnerabilities during the process of upgrade, we have published partial mitigations (see question 6 below), as additional security controls to reduce security exposure in the interim, and will continue to update guidance as applicable. 

11. Why is SVD-2022-0608 an exception to the decision to backport to 8.x.1 / 8.x.2?
    SVD-2022-0608, is the single critical vulnerability released with the June 14th Security Advisory that cannot be mitigated without turning off the deployment server, and after considerable analysis we believe this backport has the least risk of introducing a regression. We’re in the process of releasing a backport for the versions under support (8.1.x / 8.2.x). We had initially ruled out a backport due to the risk of introducing regressions to Splunk instances that co-locate other services with the deployment server (such as license manager, deployer, etc.). We’ve performed additional work to confirm that a 9.0 deployment server is compatible with older versions of universal forwarders, license managers, etc. Customers who have a Deployment Server co-located should review the compatibility matrix in the Considerations for Upgrading section of this document to ensure the backported fix will work in your environment. We appreciate community's feedback and aim to be more responsive.
     
12. What is Splunk's vulnerability policy and its commitment to fixing and backporting vulnerabilities post 9.0?
    We remain committed to helping customers identify and remediate security issues quickly. For “Critical” or “High” vulnerabilities we plan to provide advisories and any available patches as close to real-time as possible. For vulnerabilities considered “Moderate” or “Low Risk”, we’re planning quarterly releases of any available patches so that Splunk administrators can plan for patches and upgrades on a predictable schedule. Please note we have updated our Security Advisory Policy to help clarify what Security Patch Updates are and the details about their availability.
     
13. I can’t upgrade my on-prem Splunk Enterprise deployment right now. What are my mitigations?
    Depending on the configuration or implementation, some of the vulnerabilities may not impact the instance. In addition, Splunk offers detections for some vulnerabilities. Please refer to each advisory.
    
    However, in all cases, we strongly encourage customers to update to 9.0 as soon as possible. There are no full mitigations available aside from upgrading. Partial mitigations are provided in the following table.

    Advisory	Info	Partial Mitigation
    SVD-2022-0601	Splunk Enterprise disabled TLS validation in Python 3 libraries by default	Review your apps for outbound calls that do not use TLS. Setup infra/reverse-proxy to monitor for outbound service calls. Review network layer controls. Splunk will provide a detection search that lets customers look for TLS clients that do not pass hostname or proper certificate base validation.
    SVD-2022-0602	Splunk Enterprise lacked TLS cert validation for Splunk-to-Splunk communication by default	Ensure Splunk peer communications are configured to use valid certificates.
    SVD-2022-0603	Splunk Enterprise lacked TLS hostname validation	Set infra/reverse proxy to monitor for outbound service calls. Actively review network layer controls (for example, firewall logs) to ensure hosts used in TLS connections are trusted.
    SVD-2022-0604	Risky commands warnings in Splunk Enterprise dashboards	Disable unnecessary Splunk Enterprise components and the web.conf configuration file for more information on disabling Splunk Web.
    SVD-2022-0605	Universal forwarder management services allow remote login by default	Do one of the following: Set server.conf [httpServer] disableDefaultPort = true  OR  Set server.conf [general] allowRemoteLogin = never  OR  Set web.conf [settings] mgmtHostPort = localhost:8089
    SVD-2022-0606	Splunk Enterprise and universal forwarder CLI connections lacked TLS cert validation	Ensure Splunk CLI communications are configured properly with valid certificates. Review for correct config of valid TLS certs on each Enterprise node. Keep nodes inside a VPC/VPN or network isolation.
    SVD-2022-0607	Splunk Enterprise deployment servers allow unauthenticated Forwarder bundle downloads	Disable the deployment server temporarily until it’s upgraded to 9.0.  Place the deployment server behind a firewall (be sure to allow Splunkweb and Splunkd management port). Configure pass4SymmKey on the deployment server(s) to prevent unauthenticated clients (those without the right pass4SymmKey) from connecting. Set up TLS on the deployment server(s) and configure clients to authenticate the server using transport layer security. Implement pass4SymmKey on the broker endpoint to ensure that clients are only getting serverclass information from a known, trusted Deployment Server to lower the risk of downloading undesired content from a rogue deployment server. The “handshake” operation between a deployment client and the deployment server has multiple steps, featuring two REST endpoints. The first of these is referred to as the “broker” endpoint, which relates to the client establishing its connection, and being placed into a serverclass. This portion of the client/server relationship can be equipped with mutual authentication (pass4SymmKey) with versions of Splunk Enterprise prior to 9.0. However, the other significant endpoint involved in this operation (called the “streams” or “streams:deployment” endpoint) is used by clients to download configuration apps from the deployment server. This endpoint does not support mutual authentication (pass4SymmKey) in versions of the deployment client prior to 9.0.  Implement certificate validation of the deployment server’s certificate as a mitigating control.
    SVD-2022-0608	Splunk Enterprise deployment servers allow client publishing of Forwarder bundles	Disable the deployment server temporarily until you update to 9.0. Place the deployment server behind a firewall and reduce the reachability of the DS to specific IPs or network segments where possible.

Splunk Cloud Platform deployments

1. Is Splunk Cloud Platform impacted by these vulnerabilities? 
   Yes, SVD-2022-0601, SVD-2022-0602, and SVD-2022-0603 impact all Splunk Cloud Platform instances as of Jun 14, 2022. SVD-2022-0604 impacts a minority of Splunk Cloud Platform instances on 8.2.2106 and earlier.
    
2. When will Splunk update my Splunk Cloud Platform instance and enable the fixes?
   Due to the complexity and potential impact of fully remediating a cloud deployment, roll out requires careful planning and coordination as to not disrupt customers. We aim to patch all customer cloud deployments aligned to our 4-6 weekly iterative release process by early September. 
    
3. How can I tell in Splunk Cloud Platform if I have been upgraded?
   For help identifying your Splunk Cloud Platform versions, please refer to the Splunk Docs Troubleshooting Manual. However, even if your deployment is on the 8.2.2203.x release, Splunk must perform additional steps to configure and enable the changes. 
    
4. How will a Splunk Cloud Platform user know when configurations have been implemented? 
   These vulnerabilities will be patched during the upgrade process to Splunk Cloud Platform version 9.0.2205. If you are on a lower version of Splunk Cloud Platform, you must first be upgraded to 9.0.2205. Note that this upgrade does require a maintenance window. If you wish to verify the state of the feature flags you can run the following SPL query in your Splunk Cloud Platform environment.

   Splunk Cloud Platform Configurations SPL

   The following SPL allows a Splunk Cloud Platform customer to determine the enforce TLS configurations after updating to 8.2.2203. 

   | rest /services/configs/conf-server/kvstore  ```Returns the configuration of KVStore TLS Enforcement & if the config is set in /system or an app```
   | eval sslVerifyServerCert= if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert),  kvstore_configuredSystem=if(app_dir="system","true","false")  
   | stats values(sslVerifyServerCert) as kvstore_sslVerifyServerCert values(eai:acl.app) as kvstore_configuredApp by splunk_server 
   | eval kvstore_configuredSystem=if(kvstore_configuredApp="system","true",kvstore_configuredApp) 
   | fields kvstore_sslVerifyServerCert, splunk_server, kvstore_configuredSystem
   | append [
     | rest /services/configs/conf-server/sslConfig ```Returns the current configuration for enforcement of TLS Hostname Validation & if the config is set in /system or an app```
     | eval sslVerifyServerName=if(isnull(sslVerifyServerName),"unset",sslVerifyServerName) | stats values(sslVerifyServerName) as servername_sslVerifyServerName values(eai:acl.app) as servername_configuredApp by splunk_server 
     | eval servername_configuredSystem=if(servername_configuredApp="system","true",servername_configuredApp)     
     | fields servername_sslVerifyServerName, splunk_server, servername_configuredSystem
   ]
   | append [
     | rest /services/configs/conf-server/sslConfig ```Returns the current configuration for enforcement of TLS Certificate validation for Splunk-to-Splunk communications & if the config is set in /system or an app```
     | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert) 
     | stats values(eai:acl.app) as global_configuredApp values(sslVerifyServerCert) as global_sslVerifyServerCert by splunk_server 
     | eval global_configuredSystem=if(global_configuredApp="system","true","false") 
     | fields global_sslVerifyServerCert, splunk_server, global_configuredApp 
   ]
   | append [ 
     | rest /services/configs/conf-server/pythonSslClientConfig ```Returns the current configuration for enforcement of Python TLS Certificate Validation & if the config is set in /system or an app - note that this can also be set in splunk-launch.conf but we cannot use search to read that config```
     | eval sslVerifyServerCert=if(isnull(sslVerifyServerCert),"unset",sslVerifyServerCert)
     | stats values(eai:acl.app) as python_configuredApp values(sslVerifyServerCert) as python_sslVerifyServerCert values(sslVerifyServerName) as python_sslVerifyServerName by splunk_server 
     | eval python_configuredSystem=if(python_configuredApp="system","true","false")
     | fields python_sslVerifyServerCert, python_sslVerifyServerName, splunk_server, python_configuredSystem
   ]
   | stats values(*) as * by splunk_server

   The output will look similar to the following:

   

   Note that values "unset" and "0" mean that you are still in "warning mode" for that particular feature flag:

   • "global_sslVerifyServerCert" for cert validation
   • "kvstore_sslVerifyServerCert" for KVStore cert validation
   • "servername_sslVerifyServerName" for TLS Hostname validation

   For Python TLS, there is no way to query this from the search bar completely, but the "python_sslVerifyServerCert" & "python_sslVerifyServerName" indicates the values for server.conf the stanza [pythonSslClientConfig]. However, you can also set this in splunk-launch.conf as well, which is not searchable via the | rest command in SPL.

5. Regarding “Python TLS enforcement may break 3rd party and customer apps”, is this potential break on the mitigation or upgrade of the CMP components of Splunk Cloud Platform customers or when Splunk Cloud Platform upgrades to 8.2.2203 or both?
   For CMP customers, private and third party apps may be impacted when the settings under "Configure TLS host name validation for Splunk Python modules" are turned on.  For Splunk Cloud Platform, we plan to enable/enforce the configuration during the 9.0.2205 release train and upgrades.

Next steps

• Read more about upgrades on Splunk Answers:
  • How do I benchmark system health before a Splunk Enterprise upgrade?
  • How do I monitor system health during a Splunk Enterprise upgrade?
  • What do I validate after I upgrade Splunk Enterprise to confirm the upgrade was successful?
• Continue the discussion about Splunk Enterprise upgrades, get your questions answered and connect with your peers on Splunk Community.
• Review the following resources:
  • Product Security Page. Subscribe to get notified of all recent advisories. 
  • Documentation. Read the detailed steps on how to take action. 
  • Improve Your Security Posture Tech Talk. Technical webinar focusing on our Splunk Enterprise 9.0 security features and June 2022 security advisories.
  • Contact Customer Success. General contact form if you do not have your specific manager’s contact.
  • OnDemand Services. Email if you cannot open a request or do not know if you have OnDemand access.