Skip to main content
Splunk Lantern

Filesystem mounts after *nix patching event

You might want to count the number of mounted directories before and after patching when doing the following:

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

Example

System patching is a risky process in a production environment. Based on the method by which the directory was originally mounted, it might not survive the patching and reboot event. You want a search that lets you determine the number of mounted directories before and after a patching event so you can validate the state of the system.

To optimize the search shown below, you should specify an index and a time range.

  1. Run the following search:
sourcetype="df" earliest=-15m@m latest=now 
|eval dataset="last 15m" 
|append 
[| search index=<index name> sourcetype="df" earliest=-75m@m latest=-60m@m 
|eval dataset="1h ago"]
|stats dc(dataset) AS dc_dataset values(dataset) AS values_dataset BY filesystem mount host 
|eval no_longer_mounted = if(dc_dataset=1 AND values_dataset="1h ago", filesystem." -> ".mount, null()) 
|eval newly_mounted = if(dc_dataset=1 AND values_dataset="last 15m", filesystem." -> ".mount, null())
|eval still_mounted = if(dc_dataset=2, filesystem." -> ".mount, null())
|stats values(*_mounted) AS *_mounted BY host

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="df" 

Search disk space on mounted volumes. 

earliest=-15m@m latest=now

Search for events occurring in the last 15 minutes

|eval dataset="last 15m" 

Set the field named dataset to the quoted string.

|append 

[| search index=<index name> sourcetype="df" earliest=-75m@m latest=-60m@m 

|eval dataset="1h ago"]

Search for events that occurred an hour ago and look back for 75 minutes. Append the results to the primary search.

|stats dc(dataset) AS dc_dataset values(dataset) AS values_dataset BY filesystem mount host 

Get a distinct count of the dataset, grouped by filesystem, and put the contents of the dataset into values.

|eval no_longer_mounted = if(dc_dataset=1 AND values_dataset="1h ago", filesystem." -> ".mount, null()) 

Create the no_longer_mounted field for a directory distinct count of “1” when the count only existed 1 hour ago. 

|eval newly_mounted = if(dc_dataset=1 AND values_dataset="last 15m", filesystem." -> ".mount, null())

Create the newly_mounted field for a directory distinct count of “1” when the count only exists within the last 15 minutes. 

|eval still_mounted = if(dc_dataset=2, filesystem." -> ".mount, null())

Create the still_mounted field for a directory distinct count of “2” when the count only existed 1 hour and 15 minutes ago. 

|stats values(*_mounted) AS *_mounted BY host

Create a list of mounted statuses for each directory by host.

Result

Use this procedure to validate that the number of mounted directories is the same before and after the patching event. If the number of directories changed, the system integrity might be compromised and you should perform additional validation. 

Sample results for this search are shown in the table below.

host no_longer_mounted still_mounted

ip-172-31-64-114.ec2.internal

/dev/xvdf -> /extappstorage

/dev/xvda1 -> /

ip-172-31-71-164.ec2.internal

 

/dev/xvda1 -> /

ip-172-31-79-80.ec2.internal

 

/dev/xvda1 -> /

  • Was this article helpful?