Skip to main content
Splunk Lantern

Kubernetes service accounts with failed or forbidden status

You might want information about which Kubernetes service accounts received failed or forbidden status during access attempts when doing the following:

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

Example

You want to investigate failed or forbidden status accounts on your network to determine if they represent a threat. 

To optimize the search shown below, you should specify an index and a time range.

AWS

  1. Ensure that your deployment is ingesting CloudWatch logs.
  2. Run the following search: 
sourcetype="aws:cloudwatchlogs:eks" user.groups{}=system:serviceaccounts responseStatus.status=Failure 
| table sourceIPs{} src_user userAgent verb responseStatus.status requestURI 
Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="aws:cloudwatchlogs:eks" 

Search only AWS EKS Kubernetes data.

user.groups{}=system:serviceaccounts 

Search the service accounts user group.

responseStatus.status=Failure

Search for requests with a failure status.

| table sourceIPs{} src_user userAgent verb responseStatus.status requestURI 

Display the results in a table with columns in the order shown.

Azure

  1. Ensure that you have configured Kube-Audit data diagnostics.
  2. Run the following search: 
sourcetype=mscs:storage:blob:json category=kube-audit 
| spath input=properties.log 
| search user.groups{}=system:serviceaccounts*  responseStatus.reason=Forbidden 
| table  sourceIPs{} user.username userAgent verb responseStatus.reason responseStatus.status properties.pod objectRef.namespace  

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype:mscs:storage:blob:json 

Search only the source type mscs:storage:blob:json. 

category=kube-audit

Search the data source kube-audit from the diagnostic logs in Azure Cloud services.

| spath input=properties.log 

Extract fields from the properties Kube-Audit log.

| search user.groups{}=system:serviceaccounts*  responseStatus.reason=Forbidden 

Search for service accounts with a response status of Forbidden.

| table  sourceIPs{} user.username userAgent verb responseStatus.reason responseStatus.status properties.pod objectRef.namespace  

Display the results in a table with columns in the order shown.

GCP

  1. Ensure that your deployment is ingesting Pub/Sub messaging logs.
  2. Run the following search:
sourcetype="google:gcp:pubsub:message" system:serviceaccounts data.protoPayload.response.status.allowed!=* 
| table src_ip src_user http_user_agent data.protoPayload.response.spec.resourceAttributes.namespace data.resource.labels.cluster_name data.protoPayload.response.spec.resourceAttributes.verb  data.protoPayload.request.status.allowed data.protoPayload.response.status.reason data.labels.authorization.k8s.io/decision 
| dedup src_ip src_user 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype="google:gcp:pubsub:message" 

Search only GCP Pub/Sub messages.

system:serviceaccounts data.protoPayload.response.status.allowed!=*

Search for service accounts that do not have a response status of allowed.

| table src_ip src_user http_user_agent data.protoPayload.response.spec.resourceAttributes.namespace data.resource.labels.cluster_name data.protoPayload.response.spec.resourceAttributes.verb  data.protoPayload.request.status.allowed data.protoPayload.response.status.reason data.labels.authorization.k8s.io/decision 

Display the results in a table with columns in the order shown.

| dedup src_ip src_user 

Remove duplicate results from the same IPs and users.

Result

You can extend this search by using top or rare operators to find trends or rarities in failure status, user agents, source IP addresses, and request URIs. Note that this search can give false positives as there might be inherent issues with authentications and permissions at cluster.

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub for AWS, Azure, or GCP.

  • Was this article helpful?