Securing the Splunk platform with TLS

Issuer certificates & certificate authorities

These are your root and intermediate certificates. They don’t represent a system or set of systems; instead they represent a trusted authority on verification. The terms “issuer certificate” and “certificate authority” can be used interchangeably.

• A root certificate is most often used to issue intermediate certificates, and then the intermediate is used to issue leaf (client/server) certificates. This improves security; however, intermediates are not required. A root certificate can issue a leaf certificate directly, and this is allowed in Splunk platform.
• “Issuing” and “signing” are done using Certificate Signing Requests (CSRs).
• In Splunk platform, when TLS verification is enabled, the system will refer to its “CA Bundle”. This is a single file (sslRootCAPath) that contains all your issuer certificates. The bundle initiates a connection to a remote system (A -> B) with sslVerifyServerCert or to respond to a remote system that initiated the connection (A <- B) with requireClientCert. The verification works by checking the issuers of the presented certificate and seeing if they are in the CA bundle.

Leaf (client/server) certificates

Leaf means that the certificate is unable to sign any additional certificates. They are often referred to as client or server certificates because that’s generally what they represent, but these are not technical TLS terms.

• Splunk platform systems use server certificates, meaning the certificate should represent the system(s) in the Subject Alternative Name (SAN) line and Common Name (CN) value. Splunk platform allows wildcard CN/SANs to be used. You can also put multiple hosts in the SAN, but this can become difficult to manage or update compared to a wildcard.
• Universal forwarders (or web browsers, if desired) use client certificates. These are called client certificates because they don’t need to represent (the CN/SAN) the system they’re installed on. They only need to be signed by an issuer that the Splunk platform server trusts. You might hear these referred to as forwarder certificates in the Splunk ecosystem.
• You’ll also often hear the term “(full) certificate chain” when reading about TLS. A certificate chain is a leaf certificate that has the proper issuer certificates under it in a single file. In Splunk we automatically create the chain by using the client/serverCert and sslRootCAPath values automatically, so you should not create a "full chain certificate". You should place the server/client certificate and private key in one file, and all of your issuer certificates in another file.

Certificate Signing Request (CSR)

A CSR contains information about a specific server or subset of servers. The CSR must be signed by an issuer certificate in order to become a leaf (client/server) certificate. When generating the CSR, there are a few important items to be aware of:

• The private key used in the CSR is required only on the system that the certificate is for (not the issuer, not clients connecting to it, etc). Do not lose this, because the certificate won’t work without it. Also, do not put a passphrase on the private key if you plan to use requireClientCert=true!
• The CN of a server certificate cannot match the CN of its issuer certificates. Matching is called being self-signed and will make the certificate invalid.
• The date range must be valid. Check the Not Before and Not After fields displayed on the certificate.
• The Subject Alternative Name(s) (SAN) are crucial for proper implementation if you turn on TLS verification. In Splunk platform, the SAN is used to verify that who you are connecting to is who the certificate is actually for. Whatever you put in your Splunk platform configuration files (like targetUri type settings), in your browser, or in a command line must be in the SAN for a valid and verified TLS connection. A SAN can include one or all of the following:
  • Hostnames. Generally shortname or FQDN of the system.
  • DNS aliases. If you have any DNS records that may be used to refer to the system.
  • Wildcards. Generally the domain of your Splunk systems. You can have more than one wildcard if needed. For each wildcard, only the leftmost subdomain can be wildcarded.
    • Correct: *.example.com - This will not work if the hostname is something like host.sub.example.com. The proper wildcard for that would be *.sub.example.com.
    • Incorrect: host.*.example.com - This won’t work because the wildcard must be at the start.
  • IP. Any IP address that can be used to connect to the system.

PEM certificates

All certificates in the Splunk platform must be in PEM format. If you receive a different certificate format from your PKI team, you can usually convert these to PEM with the openssl command. You can find this using any search engine with a string like openssl convert X to pem.

Here’s an example of what PEM format looks like (but expect it to be much longer):

Certificate	Private Key
-----BEGIN CERTIFICATE----- MIIERTCCAi2gAwIBAgIJAO1AbuRqB79HMA0GCSqGSIb3DQECwUAMEkxCzAJBgNV BAYTAlVTMQswCQYDVQQIDAJUWDERMA8GA1UECgwITmljaBMdGQxGjAYBgNVBAMM EU5pY2sgSW50ZXJtZWRpYXRlMB4XDTIyMDcyMjE1MDMxM1XDTIzMDcyMjE1MDMx -----END CERTIFICATE-----	-----BEGIN PRIVATE KEY----- MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDraWi9fysB2Tf5 QODur7b5i2ZWJ407zjnp9YXHxu+Ecs4i15iJ0SYC1v4rG7nSZihlSo2rRFP07ZRu wEfYKkayWWkn2o/Ip+M8VCXlCbtrOl26YWFDXPyec4opI4+RuDvytgKbNum1D5N -----END PRIVATE KEY-----

Test TLS connectivity

To check basic connectivity, this command is usually your first go-to. You may want to include the -quiet flag if you’d just like to see whether the connection is having issues. You likely need to pass the -CAfile parameter, unless your instance has a publicly trusted TLS certificate.

openssl s_client -connect host:port
openssl s_client -CAfile /path/here/cabundle.pem -connect host:port
openssl s_client -connect host:port -quiet

This command lets you see the actual certificates returned by the server you’re connecting to. This can be handy to make sure you see the correct certificates.

openssl s_client -connect host:port -showcerts
openssl s_client -connect host:port -showcerts -CAfile /path/here/cabundle.pem

View a certificate

Sometimes you might need to “view” a certificate so that you can see details such as the subject, issuer, SAN, CN, and expiration. This is the equivalent of clicking the little lock icon in your browser.

openssl x509 -noout -text -in cert.pem

Certificates typically have several fields that determine the validity of that certificate. Typical fields are:

• Version. Version number.
• Issuer. The entity (typically a Certificate Authority (CA)) that issued the certificate. In order for a certificate to be trusted, its issuer must be trusted.
• Signature. The signature of the issuer.
• Subject. The entity to which this certificate applies. In the case of TLS certificates for websites, the subject is an LDAP-like string, featuring a CN, or Common Name, of the site providing the certificate. Compare the CN field with the Subject Alternative Name (SAN).
• Validity. Start and end dates for the validity of the certificate. If the current date is not within the validity range of the certificate, it is considered invalid.

Verify private key and certificate match

In order for your Splunk platform instance to be able to decrypt traffic sent to it, it needs to have the correct private key. When implementing TLS, it’s critical to check that you have the correct private key for your certificate. The output of these two commands must match. If your instance doesn’t allow you to run the md5 command, simply remove the last pipe from these commands.

openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in privkey.key | openssl md5

Build your certificate chain

1. Take all of the certificate blocks you have and place them into separate files. They’re only text, so you don't need any programs other than a text editor.
2. Run the following command against each of the separated certificate files: openssl x509 -noout -issuer -subject -in somecert.pem
3. Find the certificate with the subject that matches the server you’re working on. This is your leaf (client/server) certificate.
4. Find the certificate where the subject matches the issuer of the leaf certificate identified in the previous step.
   • If the issuer and subject of this certificate are the same, then you signed the leaf certificate with a root certificate. This isn’t recommended, but it is allowed. Go to step 6.
   • If the issuer and subject of this certificate are different, then you’ve identified the intermediate certificate.
5. Find the certificate where the subject matches the issuer of the intermediate certificate identified in the previous step. This is the root certificate, which means you have a complete chain. If the issuer and subject are different, then you either:
   • have multiple intermediates (this is fine - repeat this process until you reach the root certificate)
   • are missing the root certificate (in which case, you cannot proceed until you have it)
6. Put the leaf certificate followed by the private key for that leaf into a single file. There should only be two blocks of text inside this file, meaning do not include issuer certificates. This file is your clientCert or serverCert.
7. Combine the intermediate(s) and root into a single file. Put these in order based on what you discovered in step 5. Typically you’ll have an intermediate followed by the root and nothing else. If you have multiple intermediates, then put them in order. In other words, if X issued Y and Y issued Z, then the order is Z at the top, Y in the middle, and X at the bottom. No private keys or leafs should go in this file. This is your sslRootCAPath.
8. To validate this programmatically, refer to Verify your certificate chain.

   

Verify your certificate chain

After you’ve identified that you have all of the correct certificates, next you’ll verify that the chain is complete and in the correct order.

1. If you don’t still have all of the certificates in separate files, split them up into their own files.
2. Make sure that the root certificate trusts the intermediate. If you have multiple intermediate certificates, you may want to repeat this for those, but this command can let you know if there is a problem: openssl verify -CAfile root_ca.pem intermediate_ca.pem
3. If step 2 validates as expected, then use the CA bundle from what you created in Build your certificate chain to verify the entire chain trusts the leaf certificate: openssl verify -CAfile ca_certs.pem server.pem

Connection with OpenSSL hangs

If openssl s_client hangs before it can report anything about certificates, there’s a problem with the TLS server. In most instances, this is because the sslPassword doesn’t match the one that was used to generate the key. Check this first. The splunkd.log will say something like this:

07-15-2022 22:41:26.142 +0000 ERROR SSLCommon [18692 HTTPDispatch] - Can't read key file /opt/splunk/etc/apps/mgmt_secure_server/auth/octopus_sh-mgmt_server.pem SSL error code=101077092 message="error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt"

Unable to get local issuer

This warning message indicates that the TLS client (in this case, OpenSSL) couldn't identify who issued the certificate. That means that the CA identified in the certificate is unknown. If you get this message in Splunk platform, you’re most likely missing the appropriate root/intermediate certificates in the sslRootCAPath. However, if you’re seeing this with the openssl command, you likely need to pass the -CAFile argument:

# openssl s_client -connect 12.23.45.6:8000 -quiet -CAfile $SPLUNK_HOME/etc/auth/cacert.pem.default 
depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
verify return:1
depth=0 CN = cm.so.splunklabs.net, O = SplunkUser
verify return:1

Self-signed certificate in chain

As part of the output from an attempt to openssl s_client, this message indicates that one of the certificates within the chain is self-signed. That is, the subject of the certificate is the same as the issuer. In order for OpenSSL to validate this, the self-signed certificate must be explicitly trusted by using the -CAfile option. The equivalent in Splunk platform would be adding this to the sslRootCAPath.

Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=Splunk Inc/OU=Cloudworks/CN=*.cloud-architects.splunkcloud.com
   i:/C=US/ST=California/L=San Francisco/O=Splunk Inc/OU=Cloudworks/CN=Splunk Cloud Certificate Authority
 1 s:/C=US/ST=California/L=San Francisco/O=Splunk Inc/OU=Cloudworks/CN=Splunk Cloud Certificate Authority
   i:/C=US/ST=California/L=San Francisco/O=Splunk Inc/OU=Cloudworks/CN=Splunk Cloud Certificate Authority

Lines 4-5 show that the signing certificate (the issuer) for the leaf certificate on line 2, has the same subject as issuer, indicating that this is a self-signed certificate.

Key values mismatch

This message occurs when the private key for a certificate doesn’t match the certificate itself. It can occur most frequently with a misconfiguration (for example, the wrong key file is configured), but could be a problem with the certificate generation process. The underlying principle here is that the public side of the key is algorithmically linked to the private key that the server has. If the math doesn’t work, then it’s the wrong key. The openssl command can be used to validate the keys of a certificate / key pair:

openssl x509 -noout -modulus -in <CERTIFICATE> | openssl md5
openssl rsa -noout -modulus -in <PRIVATE_KEY> | openssl md5

Prerequisites

Before you get started you will need:

1. A certificate for each Splunk Enterprise or intermediate universal forwarder server that has a valid CN and SAN list. This means that the same certificate will be used for all TLS functions (Web/Forwarding/Management) for that given host. You can use a wildcard certificate or put multiple server names in a single certificate.
2. A single client (“forwarder”) certificate that is signed by a CA you will include in your CA bundle (see View a certificate).
3. The private keys and passphrases (if used) used during the CSR for the above certificates.
4. All of the root and intermediate certificates used for signing the above certificates.

Example environment

• 100 universal forwarders with various names
• 2 intermediate universal forwarders (iuf01, iuf02)
• 1 search head (sh01)
• 1 deployment server (ds01)
• 5 indexers (idx01…05)
• 1 monitoring console (mc01)
• All managed Splunk systems in the splunk.buttercup.org domain
• One internal root and intermediate authority

Here are some examples of what this could look like, but in practice you can mix-and-match however you see fit.

Unique certificates: 10 server certificates, 1 root, 1 intermediate, 1 client certificate

CN = iuf01
SAN = iuf01, iuf01.splunk.buttercup.org, 172.16.0.1

CN = sh01
sh01, sh01.splunk.buttercup.org, 172.16.0.3

Wildcard certificates: 1 server certificate, 1 root, 1 intermediate, 1 client certificate

CN = *.splunk.buttercup.org
SAN = *.splunk.buttercup.org

All host certificates: 1 server certificate, 1 root, 1 intermediate, 1 client certificate

CN = ManagedSplunkServers
SAN = iuf01, iuf01.splunk.buttercup.org, 172.16.0.1, iuf02, iuf02.splunk.buttercup.org, 172.16.0.2, sh01, sh01.splunk.buttercup.org, 172.16.0.3, ds01, ds01.splunk.buttercup.org, 172.16.0.4, etc, etc 

Tiered host certificates: 5 server certificates, 1 root, 1 intermediate, 1 client

CN = Indexers
SAN = idx01, idx01.splunk.buttercup.org, 172.16.0.5, idx02, idx02.splunk.buttercup.org, 172.16.0.6, … idx05, idx05.splunk.buttercup.org, 172.16.0.9, 

CN = IntermediateForwarders
SAN =  iuf01, iuf01.splunk.buttercup.org, 172.16.0.1, iuf02, iuf02.splunk.buttercup.org, 172.16.0.2

Splunk TLS configuration parameters explained

Here are some simplified definitions for the various parameters used when setting up TLS in Splunk platform:

• serverCert - When I receive a connection, this is the certificate I will present to the client.
• clientCert - When I establish a connection, this is the certificate I will present to the remote server.
• sslRootCAPath - This is the collection of issuer certificates that I trust. I'll refer to this bundle when I’m asked to validate a TLS connection.
• sslVerifyServerCert - When I establish a connection, I’ll check if whoever issued the certificate being presented to me is also in my sslRootCAPath. I will not verify if they are who they say they are.
• sslVerifyServerName - When I establish a connection, I’ll check if how I’m connecting to the server is on the SAN list of the certificate being presented to me. I will also challenge them to prove they are who they say they are.
• requireClientCert - When I receive a connection, I'll require the client to present a certificate. I’ll check if whoever issued the certificate being presented to me is also in my sslRootCAPath. I will not verify if they are who they say they are.
• ssl[Common/Alt]NameToCheck - When I establish a connection, I'll look at the [CN/SAN] of the certificate being presented to me and check if it matches the [CN/SAN] I require. I will not verify if they are who they say they are.

Information for Splunk Cloud Platform users

In order to send data to Splunk Cloud Platform, the connection is secured with TLS in the 100_x_splunkcloud app (also referred to as the UF Credential Package. This app comes with a server.conf file that sets sslRootCAPath. This means that if you follow the instructions below and don’t account for this, you will either break your data forwarding or break your internal TLS communications.

You must consider app name and default versus local precedence when you are configuring your environment. You also need to keep in mind that the Splunk Cloud Platform certificates expire, which means at some point you’ll need to remember how you set this up and be able to replace the necessary certificates.

The Splunk Cloud Platform outputs app has sslRootCAPath set in default/server.conf. It's recommended to name your app 000_uf_tls and set sslRootCAPath in local/server.conf in this new app, as this will ensure that it wins out over the Splunk Cloud Platform app. When structuring the file that you point sslRootCAPath to, the intermediate certificates go above the root certificates. Additionally, you can put comments with a hash mark (#Like this) above each certificate block so you can more easily keep track of what each certificate block is for. Your final CA combination should have both the Splunk Cloud Platform intermediate/root certificates (100_x_splunkcloud/default/stack_cacert.pem) and your own root/intermediate certificates inside of a single file that the 000_uf_tls app points to in local/server.conf. If done correctly, you will be able to secure your on-premise Splunk communications and also securely forward to Splunk Cloud Platform without any issues.

1. Review configuration notes

Within this process, note that the file paths for serverCert and clientCert and their applicable certs will vary depending on what type of certificates you use.

• If you use a wildcard certificate or a certificate that has all the Splunk hosts on it, then you can include it inside of an app and deploy it to the applicable systems. This is because all hosts will all be set to the same path and certificate.
• If you’re using unique certificates, you cannot put those parameters or the certificates in an app. This is because you can only provide one value to the parameter and each server will have a different certificate and a different path to that certificate. You must set serverCert/clientCert locally on the system and also place the certificate locally. It is possible to deploy those two parameters if you set the value to something that is not inside of the app itself, for example, $SPLUNK_HOME/etc/auth/server_cert.pem. You will still need to place the certificate file there locally.

If you are required to use a passphrase on your private keys, sslPassword needs to be set in $SPLUNK_HOME/etc/system/local/server.conf and cannot be handled within an app. This is specifically for the sslPassword set under sslConfig of server.conf. Additionally, you cannot have a passphrase on the private key of a client if you set up requireClientCert=true on a remote server, as the remote host will refuse to communicate with the client. For example, the deployment server will refuse to communicate with a universal forwarder. Enterprise systems will not be able to start up their web service, because Splunk Web is a client of Splunkd. Keep in mind that adding a passphrase greatly complicates administration of TLS (and again, doesn't work with requireClientCert) and anyone who has access to view the private key most likely also has access to decrypt the passphrase using splunkd and therefore access the private key anyway.

You can change any app or file names used within this process. The names used here are suggestions and not requirements. You are responsible for understanding app name precedence.

Finally, if you use the Splunk UI to install apps or check for app updates, you'll also need to append the contents of $SPLUNK_HOME/etc/auth/appsCA.pem to your CA bundle used for sslRootCAPath.

2. Validate your certificates and files

Before you proceed with any TLS implementation, it is critical to perform the following verification to ensure you have the correct certificates and that everything is valid. If something fails at any one of these steps, do not proceed. Go back to the relevant areas of this guide and make changes to ensure that failures do not occur before you move ahead.

1. View the certificates. Check that the CN/SAN looks right and that the certificate has the appropriate before/after. Do this for leaf and CA certificates.
2. Verify that the private key and certificate match for every client/serverCert you will put in place.
3. Combine your certificates and keys into the appropriate files.
4. Validate the certificate chain.

3. Set up TLS on Splunk Enterprise servers

To complete this process, you must have validated your certificates and files, reviewed the process notes, and have all necessary certificates and private keys.

There is no TLS verification enabled by default in Splunk platform. For the first phase of setup, only put certificates in place for Splunkd and Splunkweb. Do not enable verification until after TLS certificates have been deployed everywhere and you’ve verified Splunk is functioning as expected.

1. Create an app folder called 000_enterprise_tls. Create a certs and local folder within this newly created folder.
2. Combine the server certificate and private key into a single file and place it inside 000_enterprise_tls/certs/server_cert.pem. Do not include the root/intermediates in this file, there should be only two blocks of text in this file. The private key goes below the server certificate.
3. Place the private key into a single file and place it inside 000_enterprise_tls/certs/server_cert.key.
4. Combine your root and intermediate certificates and place it inside 000_enterprise_tls/certs/ca_certs.pem with intermediates at the top, and roots at the bottom.
5. In 000_enterprise_tls/local/server.conf, set the following:

   [sslConfig]
   serverCert= $SPLUNK_HOME/etc/apps/000_enterprise_tls/certs/server_cert.pem
   sslRootCAPath = $SPLUNK_HOME/etc/apps/000_enterprise_tls/certs/ca_certs.pem
   

6. In 000_enterprise_tls/local/web.conf, set the following:

   [settings]
   enableSplunkWebSSL = true
   serverCert= /opt/splunk/etc/apps/000_enterprise_tls/certs/server_cert.pem
   privKeyPath = /opt/splunk/etc/apps/000_enterprise_tls/certs/server_cert.key
   

   Do not use $SPLUNK_HOME in web.conf. You must use an absolute path and change accordingly.

7. Restart the Splunk platform. Validate that you’re able to get to the web UI and log in.
   • If you don’t receive a 500 error when logging into Splunk Web then your syntax is correct.
   • If you do see an error, then there’s an issue with one of the steps above. Look for X509Verify and SSLCommon messages in the splunkd.log for more detail.

4. Set up TLS on universal forwarders

To complete this process, you must have validated your certificates and files, reviewed the process notes, and have all necessary certificates and private keys.

For the universal forwarders, you should use a deployment server or some centralized tool such as Ansible or SCCM since they all use the same configuration. This initial setup does secure the universal forwarders. It is only a prerequisite for additional security that will be enabled later.

5. Set up TLS for Splunk to Splunk receiving/forwarding

To complete this process, you must have Set up TLS on Splunk Enterprise servers and Set up TLS on universal forwarders.

This setup is the minimum needed to turn on TLS data forwarding. Verification settings will be implemented in a later phase.

Deployment server connections

To complete this process, you must have Set up TLS on Splunk Enterprise servers and Set up TLS on universal forwarders.

TLS forwarding

To complete this process, you must have Set up TLS for Splunk to Splunk receiving/forwarding.