Skip to main content
Lantern Home
Splunk Lantern

Suspicious kubectl calls (Azure Kubernetes Service)

  1. Last updated
  2. Save as PDF

Kubectl calls are not malicious by nature. However, examining the source IP addresses, source users, user agents, object paths, and authorizations of these calls can reveal potentially malicious activity, especially if you find anonymous, suspicious IP addresses trying to access sensitive objects, such as configmaps or secrets. You want to investigate anonymous kubectl calls on your network to determine if they represent a threat. 

Required data 

Procedure

  1. Ensure that you have configured Kube-Audit data diagnostics.
  2. Run the following search. You can optimize it by specifying an index and adjusting the time range.  
sourcetype=mscs:storage:blob:json category=kube-audit 
|spath input=properties.log 
|spath input=responseObject.metadata.annotations.kubectl.kubernetes.io/last-applied-configuration 
|search userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 
|table sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI 
|rare sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

sourcetype:mscs:storage:blob:json 

Search only the source type mscs:storage:blob:json. 

category=kube-audit

Search the data source kube-audit from the diagnostic logs in Azure Cloud services.

| spath input=properties.log 

Extract fields from the properties Kube-Audit log.

| spath input=responseObject.metadata.annotations.kubectl.kubernetes.io/last-applied-configuration 

Return the field values of Kubectl calls. 

| search userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 

Find the wildcard string kubectl* to reveal the use of kubectl application, which carries out HTTP requests to the Kubernetes API. Exclude a legitimate source IP address or range of addresses from the search.

| table sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI 

Display the results in a table with columns in the order shown. 

| are sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace 

Display the least common source IP addresses, requests verbs, user agents, user groups, targeted resources, and namespaces.

Next steps

Kubectl is a tool that can do almost anything on a cluster, so it needs to be monitored. Unauthenticated calls indicate exposure of the API. Establishing security groups can limit API calls. Kubectl command strings can reveal malicious intent and likely access key compromise. Look at data such as geolocation, unusual users, unusual commands, request verbs, and object path.

For additional information about this search, such as its applicability to common frameworks and standards, see this project on GitHub.

Finally, you might be interested in other processes associated with the Monitoring Kubernetes sensitive object access use case.