Skip to main content
Splunk Lantern

Kubernetes accounts accessing sensitive objects

You might want information about Kubernetes accounts accessing sensitive objects, such as configmaps or secrets, when doing the following:

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

Example

Sensitive object access is not necessarily malicious, but user and object context can provide guidance for detection. You want to investigate sensitive object access on your network to determine if it represents a threat. 

To optimize the search shown below, you should specify an index and a time range.

AWS

  1. Ensure that your deployment is ingesting CloudWatch logs.
  2. Run the following search: 
sourcetype="aws:cloudwatchlogs:eks" objectRef.resource=secrets OR configmaps sourceIPs{}!=::1 sourceIPs{}!=<valid IP address>  
| table sourceIPs{} user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{} 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="aws:cloudwatchlogs:eks" 

Search only AWS EKS Kubernetes data.

objectRef.resource=secrets OR configmaps sourceIPs{}!=::1 

Search specific, sensitive resources such as secrets or configmaps.

sourceIPs{}!=<valid IP address>

Exclude a legitimate IP address from the search.

| table sourceIPs{} user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 

Display the results in a table with columns in the order shown.

| dedup user.username user.groups{} 

Remove duplicate results from the same user names and user groups.

Azure

  1. Ensure that you have configured Kube-Audit data diagnostics.
  2. Run the following search:  
sourcetype=mscs:storage:blob:json category=kube-audit 
| spath input=properties.log
| search objectRef.resource=secrets OR configmaps user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow  
| table user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{} |`kubernetes_azure_detect_sensitive_object_access_filter`

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype:mscs:storage:blob:json 

Search only the source type mscs:storage:blob:json. 

category=kube-audit

Search the data source kube-audit from the diagnostic logs in Azure Cloud services.

| spath input=properties.log
 

Extract fields from the properties Kube-Audit log.

| search objectRef.resource=secrets OR configmaps 

Search specific, sensitive resources, such as secrets or configmaps.

user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow

Search for anonymous users or any requests with allow decisions.

| table user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 

Display the results in a table with columns in the order shown.

|dedup user.username user.groups{} 

Remove duplicate results from the same user names and user groups.

GCP

  1. Ensure that your deployment is ingesting Pub/Sub messaging logs.
  2. Run the following search:
sourcetype="google:gcp:pubsub:message" data.protoPayload.authorizationInfo{}.resource=configmaps OR secrets  
| table data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name data.protoPayload.request.metadata.namespace data.labels.authorization.k8s.io/decision 
| dedup data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="google:gcp:pubsub:message" 

Search only GCP Pub/Sub messages.

data.protoPayload.authorizationInfo{}.resource=configmaps OR secrets

Search specific, sensitive resources such as secrets or configmaps.

| table data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name data.protoPayload.request.metadata.namespace data.labels.authorization.k8s.io/decision 

Display the results in a table with columns in the order shown.

| dedup data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name 

Remove duplicate results from the same caller IP address, user, and cluster name.

Result

Compromise of sensitive objects can lead to cluster compromise and further lateral movement. Secrets and configmaps are highly sensitive objects due the cluster information they contain. Multiple failed or denied attempts may indicate suspicious activity. Source IP address, users, geolocation, and reputation are other useful contextual items to evaluate to look for compromise. To refine the search, adding data.labels.authorization.k8s.io/decision!=allow may give fewer results and more specific context on errors or suspicious requests.

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.

  • Was this article helpful?