Skip to main content
Lantern Home
 
 
Splunk Lantern

Filesystem mounts after *nix patching event

  1. Last updated
  2. Save as PDF

 

System patching is a risky process in a production environment. Based on the method by which the directory was originally mounted, it might not survive the patching and reboot event. You want a search that lets you determine the number of mounted directories before and after a patching event so you can validate the state of the system.

Procedure

Run the following search. You can optimize it by specifying an index and adjusting the time range.

sourcetype="df" earliest=-15m@m latest=now 
|eval dataset="last 15m" 
|append 
[| search index=<index name> sourcetype="df" earliest=-75m@m latest=-60m@m 
|eval dataset="1h ago"]
|stats dc(dataset) AS dc_dataset values(dataset) AS values_dataset BY filesystem mount host 
|eval no_longer_mounted = if(dc_dataset=1 AND values_dataset="1h ago", filesystem." -> ".mount, null()) 
|eval newly_mounted = if(dc_dataset=1 AND values_dataset="last 15m", filesystem." -> ".mount, null())
|eval still_mounted = if(dc_dataset=2, filesystem." -> ".mount, null())
|stats values(*_mounted) AS *_mounted BY host

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="df"

Search disk space on mounted volumes.

earliest=-15m@m latest=now

Search for events occurring in the last 15 minutes

|eval dataset="last 15m"

Set the field named dataset to the quoted string.

|append

[| search index=<index name> sourcetype="df" earliest=-75m@m latest=-60m@m

|eval dataset="1h ago"]

Search for events that occurred an hour ago and look back for 75 minutes. Append the results to the primary search.

|stats dc(dataset) AS dc_dataset values(dataset) AS values_dataset BY filesystem mount host

Get a distinct count of the dataset, grouped by file system, and put the contents of the dataset into values.

|eval no_longer_mounted = if(dc_dataset=1 AND values_dataset="1h ago", filesystem." -> ".mount, null())

Create the no_longer_mounted field for a directory distinct count of “1” when the count only existed 1 hour ago.

|eval newly_mounted = if(dc_dataset=1 AND values_dataset="last 15m", filesystem." -> ".mount, null())

Create the newly_mounted field for a directory distinct count of “1” when the count only exists within the last 15 minutes.

|eval still_mounted = if(dc_dataset=2, filesystem." -> ".mount, null())

Create the still_mounted field for a directory distinct count of “2” when the count only existed 1 hour and 15 minutes ago.

|stats values(*_mounted) AS *_mounted BY host

Create a list of mounted statuses for each directory by host.

Next steps

Use this procedure to validate that the number of mounted directories is the same before and after the patching event. If the number of directories changed, the system integrity might be compromised and you should perform additional validation.

Sample results for this search are shown in the table below.

host no_longer_mounted still_mounted

ip-172-31-64-114.ec2.internal

/dev/xvdf -> /extappstorage

/dev/xvda1 -> /

ip-172-31-71-164.ec2.internal

/dev/xvda1 -> /

ip-172-31-79-80.ec2.internal

/dev/xvda1 -> /

Finally, you might be interested in other processes associated with the Maintaining *nix systems use case.