You might want information about Kubernetes accounts accessing sensitive objects, such as configmaps or secrets, when doing the following:
Prerequisites
In order to execute this procedure in your environment, the following data, services, or apps are required:
- Kubernetes
- One of the following:
- Amazon: Splunk Add-on for Amazon Web Services, Splunk App for AWS, and AWS CloudWatch data
- Microsoft: Splunk Add-on for Microsoft Cloud Services and Azure storage data
- Google: Splunk Add-on for Google Cloud Platform and Pub/Sub data
Example
Sensitive object access is not necessarily malicious, but user and object context can provide guidance for detection. You want to investigate sensitive object access on your network to determine if it represents a threat.
NOTE: To optimize the search shown below, you should specify an index and a time range.
AWS
- Ensure that your deployment is ingesting CloudWatch logs.
- Run the following search:
sourcetype="aws:cloudwatchlogs:eks" objectRef.resource=secrets OR configmaps sourceIPs{}!=::1 sourceIPs{}!=<valid IP address>
| table sourceIPs{} user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason
| dedup user.username user.groups{}
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
Splunk Search |
Explanation |
sourcetype="aws:cloudwatchlogs:eks" |
Search only AWS EKS Kubernetes data. |
objectRef.resource=secrets OR configmaps sourceIPs{}!=::1 |
Search specific, sensitive resources such as secrets or configmaps. |
sourceIPs{}!=<valid IP address> |
Exclude a legitimate IP address from the search. |
| table sourceIPs{} user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason |
Display the results in a table with columns in the order shown. |
| dedup user.username user.groups{} |
Remove duplicate results from the same user names and user groups. |
Azure
- Ensure that you have configured Kube-Audit data diagnostics.
- Run the following search:
sourcetype=mscs:storage:blob:json category=kube-audit
| spath input=properties.log
| search objectRef.resource=secrets OR configmaps user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow
| table user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason
| dedup user.username user.groups{} |`kubernetes_azure_detect_sensitive_object_access_filter`
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
Splunk Search |
Explanation |
sourcetype:mscs:storage:blob:json |
Search only the source type mscs:storage:blob:json. |
category=kube-audit |
Search the data source kube-audit from the diagnostic logs in Azure Cloud services. |
| spath input=properties.log |
Extract fields from the properties Kube-Audit log. |
| search objectRef.resource=secrets OR configmaps |
Search specific, sensitive resources, such as secrets or configmaps. |
user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow |
Search for anonymous users or any requests with allow decisions. |
| table user.username user.groups{} objectRef.resource objectRef.namespace objectRef.name annotations.authorization.k8s.io/reason |
Display the results in a table with columns in the order shown. |
|dedup user.username user.groups{} |
Remove duplicate results from the same user names and user groups. |
GCP
- Ensure that your deployment is ingesting Pub/Sub messaging logs.
- Run the following search:
sourcetype="google:gcp:pubsub:message" data.protoPayload.authorizationInfo{}.resource=configmaps OR secrets
| table data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name data.protoPayload.request.metadata.namespace data.labels.authorization.k8s.io/decision
| dedup data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
Splunk Search |
Explanation |
sourcetype="google:gcp:pubsub:message" |
Search only GCP Pub/Sub messages. |
data.protoPayload.authorizationInfo{}.resource=configmaps OR secrets |
Search specific, sensitive resources such as secrets or configmaps. |
| table data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name data.protoPayload.request.metadata.namespace data.labels.authorization.k8s.io/decision |
Display the results in a table with columns in the order shown. |
| dedup data.protoPayload.requestMetadata.callerIp src_user data.resource.labels.cluster_name |
Remove duplicate results from the same caller IP address, user, and cluster name. |
Result
Compromise of sensitive objects can lead to cluster compromise and further lateral movement. Secrets and configmaps are highly sensitive objects due the cluster information they contain. Multiple failed or denied attempts may indicate suspicious activity. Source IP address, users, geolocation, and reputation are other useful contextual items to evaluate to look for compromise. To refine the search, adding data.labels.authorization.k8s.io/decision!=allow may give fewer results and more specific context on errors or suspicious requests.
For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.
Comments
0 comments
Please sign in to leave a comment.