Suspicious kubectl calls

You might need to investigate the source of calls to your Kubernetes API when doing the following:

• Monitoring Kubernetes sensitive object access

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

• Kubernetes
• One of the following:
  • Amazon: Splunk Add-on for Amazon Web Services, Splunk App for AWS, and AWS CloudWatch data
  • Microsoft: Splunk Add-on for Microsoft Cloud Services and Azure storage data
  • Google: Splunk Add-on for Google Cloud Platform and Pub/Sub data

Example

Kubectl calls are not malicious by nature. However, examining the source IP addresses, source users, user agents, object paths, and authorizations of these calls can reveal potentially malicious activity, especially if you find anonymous, suspicious IP addresses trying to access sensitive objects, such as configmaps or secrets. You want to investigate anonymous kubectl calls on your network to determine if they represent a threat. 

NOTE: To optimize the search shown below, you should specify an index and a time range.

AWS

1. Ensure that your deployment is ingesting CloudWatch logs.
2. Run the following search: 

sourcetype="aws:cloudwatchlogs:eks" userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 src_user=system:anonymous  
| table src_ip src_user verb userAgent requestURI  
| stats count BY src_ip src_user verb userAgent requestURI 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Azure

1. Ensure that you have configured Kube-Audit data diagnostics.
2. Run the following search: 

sourcetype=mscs:storage:blob:json category=kube-audit 
|spath input=properties.log 
|spath input=responseObject.metadata.annotations.kubectl.kubernetes.io/last-applied-configuration 
|search userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 
|table sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI 
|rare sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

GCP

1. Ensure that your deployment is ingesting Pub/Sub messaging logs.
2. Run the following search:

sourcetype="google:gcp:pubsub:message" data.protoPayload.requestMetadata.callerSuppliedUserAgent=kubectl* src_user=system:unsecured OR src_user=system:anonymous 
| table src_ip src_user data.protoPayload.requestMetadata.callerSuppliedUserAgent data.protoPayload.authorizationInfo{}.granted object_path 
| dedup src_ip src_user 

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Result

Kubectl is a tool that can do almost anything on a cluster, so it needs to be monitored. Unauthenticated calls indicate exposure of the API. Establishing security groups can limit API calls. Kubectl command strings can reveal malicious intent and likely access key compromise. Look at data such as geolocation, unusual users, unusual commands, request verbs, and object path.

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub.