Most active Kubernetes service accounts
Service accounts are prime targets for attackers as they may present the opportunity to obtain credentials and move across all the resources they access. You want to investigate the most active Kubernetes service accounts for each pod to determine if their activities represent a threat.
Data required
Procedure
AWS
- Configure the Splunk Add-on for Amazon Web Services, and Splunk App for AWS.
- Ensure that your deployment is ingesting CloudWatch logs.
- Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype="aws:cloudwatchlogs:eks" user.groups{}=system:serviceaccounts objectRef.resource=pods
| table sourceIPs{} user.username userAgent verb annotations.authorization.k8s.io/decision
| top sourceIPs{} user.username verb annotations.authorization.k8s.io/decision
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
| Splunk Search | Explanation |
|---|---|
|
sourcetype="aws:cloudwatchlogs:eks" |
Search only AWS EKS Kubernetes data. |
|
user.groups{}=system:serviceaccounts |
Search the service accounts user group. |
|
objectRef.resource=pods |
Search your Kubernetes pods data. |
|
| table sourceIPs{} user.username userAgent verb annotations.authorization.k8s.io/decision |
Display the results in a table with columns in the order shown. |
|
| top sourceIPs{} user.username verb annotations.authorization.k8s.io/decision |
Show the 10 most active (by count and percentage) IP addresses, usernames, and request verbs (for example, list) with the respective decision (for example, deny or forbid). |
Azure
- Configure the Splunk Add-on for Microsoft Cloud Services.
- Ensure that you have configured Kube-Audit data diagnostics.
- Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype=mscs:storage:blob:json category=kube-audit
| spath input=properties.log
| search user.groups{}=system:serviceaccounts* OR user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow
| table sourceIPs{} user.username userAgent verb responseStatus.reason responseStatus.status properties.pod objectRef.namespace
| top sourceIPs{} user.username verb responseStatus.status properties.pod objectRef.namespace
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
| Splunk Search | Explanation |
|---|---|
|
sourcetype:mscs:storage:blob:json |
Search only the source type mscs:storage:blob:json. |
|
category=kube-audit |
Search the data source kube-audit from the diagnostic logs in Azure Cloud services. |
|
| spath input=properties.log |
Extract fields from the properties Kube-Audit log. |
|
| search user.groups{}=system:serviceaccounts* OR user.username=system.anonymous OR annotations.authorization.k8s.io/decision=allow |
Search the service account user group or anonymous access or any requests with allow decisions. |
|
| table sourceIPs{} user.username userAgent verb responseStatus.reason responseStatus.status properties.pod objectRef.namespace |
Display the results in a table with columns in the order shown. |
|
| top sourceIPs{} user.username verb responseStatus.status properties.pod objectRef.namespace |
Show the 10 most active (by count and percentage) source IP addresses, usernames, request verbs (for example, list), response statuses, pods and namespaces. |
GCP
- Configure the Splunk Add-on for Google Cloud Platform.
- Ensure that your deployment is ingesting Pub/Sub messaging logs.
- Run the following search. You can optimize it by specifying an index and adjusting the time range.
sourcetype="google:gcp:pubsub:message" data.protoPayload.request.spec.group{}=system:serviceaccounts
| table src_ip src_user http_user_agent data.protoPayload.request.spec.nonResourceAttributes.verb data.labels.authorization.k8s.io/decision data.protoPayload.response.spec.resourceAttributes.resource
| top src_ip src_user http_user_agent data.labels.authorization.k8s.io/decision data.protoPayload.response.spec.resourceAttributes.resource
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
| Splunk Search | Explanation |
|---|---|
|
sourcetype="google:gcp:pubsub:message" |
Search only GCP Pub/Sub messages. |
|
data.protoPayload.request.spec.group{}=system:serviceaccounts |
Search the service account user group. |
|
| table src_ip src_user http_user_agent data.protoPayload.request.spec.nonResourceAttributes.verb data.labels.authorization.k8s.io/decision data.protoPayload.response.spec.resourceAttributes.resource |
Display the results in a table with columns in the order shown. |
|
| top src_ip src_user http_user_agent data.labels.authorization.k8s.io/decision data.protoPayload.response.spec.resourceAttributes.resource |
Show the 10 most active (by count and percentage) source IP addresses, users, user agents, and the respective decisions on targeted resources. |
Next steps
Not all service accounts interactions are malicious. Analysts must consider IP address, verb, and decision context when trying to detect maliciousness. Service accounts are usually created for specific tasks, so a number of unusual failures, deny statuses, or forbidden statuses may indicate suspicious activity. IP addresses must be restricted by security groups or allow/deny policies, unless a cluster API must be exposed.
For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.
You might also be interested in other processes associated with the Monitoring Kubernetes sensitive role activities use case.