Kubernetes accounts accessing sensitive objects

You might want information about Kubernetes accounts accessing sensitive objects, such as configmaps or secrets, when doing the following:

• Monitoring Kubernetes sensitive object access

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

• Kubernetes
• One of the following:
  • Amazon: Splunk Add-on for Amazon Web Services, Splunk App for AWS, and AWS CloudWatch data
  • Microsoft: Splunk Add-on for Microsoft Cloud Services and Azure storage data
  • Google: Splunk Add-on for Google Cloud Platform and Pub/Sub data

Example

Sensitive object access is not necessarily malicious, but user and object context can provide guidance for detection. You want to investigate sensitive object access on your network to determine if it represents a threat. 

NOTE: To optimize the search shown below, you should specify an index and a time range.

AWS

1. Ensure that your deployment is ingesting CloudWatch logs.
2. Run the following search: 

sourcetype="aws:cloudwatchlogs:eks" objectRef.resource=secrets OR configmaps sourceIPs{}!=::1 sourceIPs{}!=<valid IP address>  
| table sourceIPs{} user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{} 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Azure

1. Ensure that you have configured Kube-Audit data diagnostics.
2. Run the following search: 

sourcetype=mscs:storage:blob:json category=kube-audit 
| spath input=properties.log
| search objectRef.resource=secrets OR configmaps user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow  
| table user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason 
| dedup user.username user.groups{} |`kubernetes_azure_detect_sensitive_object_access_filter`

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

GCP

1. Ensure that your deployment is ingesting Pub/Sub messaging logs.
2. Run the following search:

sourcetype="google:gcp:pubsub:message" data.protoPayload.authorizationInfo{}.resource=configmaps OR secrets  
| table data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name data.protoPayload.request.metadata.namespace data.labels.authorization.k8s.io/decision 
| dedup data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name 

 Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Result

Compromise of sensitive objects can lead to cluster compromise and further lateral movement. Secrets and configmaps are highly sensitive objects due the cluster information they contain. Multiple failed or denied attempts may indicate suspicious activity. Source IP address, users, geolocation, and reputation are other useful contextual items to evaluate to look for compromise. To refine the search, adding data.labels.authorization.k8s.io/decision!=allow may give fewer results and more specific context on errors or suspicious requests.

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.