Skip to main content
Lantern Home
Splunk Lantern

Kubernetes sensitive roles

  1. Last updated
  2. Save as PDF

Sensitive role resource access is necessary for cluster operation, however, you want to make sure roles with permission to sensitive objects aren't using their privileges for malicious ends. You want to investigate sensitive roles on your network to determine if they represent a threat. 

Data required 

Kubernetes data

Procedure

AWS

  1. Configure the Splunk Add-on for Amazon Web Services, and Splunk App for AWS.
  2. Ensure that your deployment is ingesting CloudWatch logs.
  3. Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype="aws:cloudwatchlogs:eks" objectRef.resource=clusterroles OR clusterrolebindings sourceIPs{}!=::1 sourceIPs{}!=<valid IP address>  
| table sourceIPs{} user.username user.groups{} objectRef.namespace requestURI annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{}

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="aws:cloudwatchlogs:eks"  

Search only AWS EKS Kubernetes data.

objectRef.resource=clusterroles OR clusterrolebindings 

Search for events from the sensitive roles, clusterrole or clusterrolebindings.

sourceIPs{}!=::1
sourceIPs{}!=<valid IP address>

Exclude a legitimate IP address or range of addresses from the search.

| table sourceIPs{} user.username user.groups{} objectRef.namespace requestURI annotations.authorization.k8s.io/reason 

Display the results in a table with columns in the order shown.

| dedup user.username user.groups{} 

Remove duplicate results from the same IPs and users.

Azure

  1. Configure the Splunk Add-on for Microsoft Cloud Services.
  2. Ensure that you have configured Kube-Audit data diagnostics.
  3. Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype=mscs:storage:blob:json category=kube-audit 
| spath input=properties.log
| search objectRef.resource=clusterroles OR clusterrolebindings 
| table sourceIPs{} user.username user.groups{} objectRef.namespace requestURI annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{}

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype:mscs:storage:blob:json 

Search only the source type mscs:storage:blob:json. 

category=kube-audit

Search the data source kube-audit from the diagnostic logs in Azure Cloud services.

| spath input=properties.log

Extract fields from the properties Kube-Audit log.

| search objectRef.resource=clusterroles OR clusterrolebindings 

Search for events from the sensitive roles, clusterrole or clusterrolebindings.

| table sourceIPs{} user.username user.groups{} objectRef.namespace requestURI annotations.authorization.k8s.io/reason   

Display the results in a table with columns in the order shown.

| dedup user.username user.groups{} 

Remove duplicate results from the same IPs and users.

GCP

  1. Configure the Splunk Add-on for Google Cloud Platform.
  2. Ensure that your deployment is ingesting Pub/Sub messaging logs.
  3. Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype="google:gcp:pubsub:message" data.labels.authorization.k8s.io/reason=ClusterRoleBinding OR Clusterrole dest=apis/rbac.authorization.k8s.io/v1 src_ip!=::1  
| table src_ip src_user http_user_agent data.labels.authorization.k8s.io/decision data.labels.authorization.k8s.io/reason 
| dedup src_ip src_user

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="google:gcp:pubsub:message" 

Search only GCP Pub/Sub messages.

data.labels.authorization.k8s.io/reason=ClusterRoleBinding OR Clusterrole 

Search for events from the sensitive roles, clusterrole or clusterrolebindings.

dest=apis/rbac.authorization.k8s.io/v1

Search events with this destination path.

src_ip!=::1

Exclude a legitimate range of addresses from the search.

| table src_ip src_user http_user_agent data.labels.authorization.k8s.io/decision data.labels.authorization.k8s.io/reason 

Display the results in a table with columns in the order shown.

| dedup src_ip src_user 

Remove duplicate results from the same IPs and users.

Next steps

If any of the following appear in your results, you should investigate further:

  • Unknown IP addresses
  • Unauthenticated or unknown usergroups that request access to sensitive roles
  • Unusual geographical location, access times, or reasons 

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.

You might also be interested in other processes associated with the Monitoring Kubernetes sensitive role activities use case.