Renewing a certificate on a new Splunk Enterprise installation

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

If you want to renew an SSL certificate - whether from Let's Encrypt or another certificate authority - on a Splunk Enterprise instance, this guide walks you through how to do this with step-by-steps and instructions on what's needed on the local machine. This guide specifically covers using Let's Encrypt with Certbot, but the steps can also be adapted for other certificate providers.

Prerequisites

To perform this process, you'll need a Splunk Enterprise web server with:

Command line access: Command-line access via SSH or directly to the web server's console.
Sudo privileges: You'll need to generate files and move them within the file system.
Splunk web server access: For validation and troubleshooting.
Certbot installed: Certbot must be installed prior to following these steps.

If Certbot isn’t installed on your server, you can install it using pip3: pip3 install certbot

Renewing existing SSL certificates

Examine the existing SSL certificates before taking any action. To look at the current certificates installed on the server, use the command `splunk cmd openssl x509 -in /path/to/hec.pem -text. This command lets you check to see if the certificate has expired and shows you other metadata.

At this point you can leave the certificate in place if no changes are needed, but it can still be replaced or rotated. This command will be used again in a later step to validate that the renewed certificate matches what you expect to see.

/opt/splunk/bin/splunk cmd openssl x509 -in 
/etc/letsencrypt/live/yourdomainname.dev/hec.pem -text

Manually updating a certificate using Certbot via the terminal

Open a terminal connection. If you do not have direct console access, then you'll need to open an SSH connection to the Splunk Enterprise host address and login.
```
ssh username@yourdomainname.dev
```
Change to the Let's Encrypt SSL files directory. This will make the next command easier so you don't need to add the full path of the files to your command.
```
cd /etc/letsencrypt/live/yourdomainname.dev/
```
Renew a Let’s Encrypt certificate using the Certbot command line tool.
```
certbot certonly --standalone -d yourdomainname.dev
```
Add a password to the private key of the certificate you are adding to your Splunk Enterprise instance. Failure to add a password to the private key will result in the web server and rest endpoints not working due to an SSL handshake error. The user private key exists in /etc/letsencrypt/live/yourdomainname.dev/
```
openssl pkcs8 -topk8 -inform PEM -outform PEM -in privkey1.pem -out privkey1.env.pem $ (interactive) password:password
```
Combine all the file parts of the certificate into a single file named hec.pem. This new hec.pem file will contain the certificate, private key (with password), and full chain.
```
 cat cert1.pem >> hec.pem cat privkey1.env.pem >> hec.pem cat fullchain1.pem >> hec.pem
```

Examine and verify the renewed certificate and its metadata by using this command.

splunk cmd openssl x509 -in /path/to/hec.pem -text. /opt/splunk/bin/splunk cmd openssl x509 -in /etc/letsencrypt/live/yourdomainname.dev/hec.pem -text

Validate that the path in your web.conf file and name are correct. A bad path can result in a failure of SSL handshakes on the web server. To do this, you'll need to use a command line text editor such as VI, VIM, or Emacs.
```
vi /opt/splunk/etc/system/local/web.conf
```

Restart Splunk Enterprise using this command.

splunk restart: sudo /opt/splunk/bin/splunk restart

Troubleshooting

Permissions errors might occur if sudo is not prepended to the noted commands, depending on your users:

Bad path or broken symbolic link: cat: etc/letsencrypt/live/yourdomainname.dev/hec.pem: No such file or directory
Validate the symbolic links in the lets encrypt certificate files: ls -ahl /etc/letsencrypt/path/to/thing
Fix a symbolic link: ln -s /path/to/link linkname
Start/stop/restart Splunk Enterprise (might require sudo): /opt/splunk/bin/splunk restart

Example configuration of web.conf note

The key value pairs might change depending on the Splunk Enterprise version:

httpport = 443
privKeyPath  = /etc/letsencrypt/live/yourdomainname.dev/privkey1.pem
serverCert = /etc/letsencrypt/live/yourdomainname.dev/cert1.pem

Next steps

These resources might help you understand and implement this guidance:

Product tip: Installing an existing certificate on a new Splunk Enterprise installation