You might need to investigate the source of calls to your Kubernetes API when doing the following:
Prerequisites
In order to execute this procedure in your environment, you may need to first on-board the services or apps shown in the following table.
|
Category |
Name |
Importance |
|
Service |
Kubernetes |
Required |
|
Add-on |
Required |
|
|
App |
Required |
|
|
Add-on |
Required |
|
|
Add-on |
Required |
Example
The purpose of this example is to show how this procedure works in a general environment. In your environment, you can optimize the search by specifying an index, a time range, or a different data source.
Kubectl calls are not malicious by nature. However, examining the source IP addresses, source users, user agents, object paths, and authorizations of these calls can reveal potentially malicious activity, especially if you find anonymous, suspicious IP addresses trying to access sensitive objects, such as configmaps or secrets. You want to investigate anonymous kubectl calls on your network to determine if they represent a threat.
AWS
- Ensure that your deployment is ingesting CloudWatch logs.
- Run the following search:
`aws_cloudwatchlogs_eks` userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 src_user=system:anonymous
|table src_ip src_user verb userAgent requestURI
|stats count BY src_ip src_user verb userAgent requestURI |`kubernetes_aws_detect_suspicious_kubectl_calls_filter`
Search Explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
|
Splunk Search |
Explanation |
|
`aws_cloudwatchlogs_eks` |
Run a macro to search AWS EKS Kubernetes data. Note: The punctuation surrounding a Splunk macro is always a back tick (`), not a single quote ('). |
|
userAgent=kubectl* |
Find the string kubectl* to reveal the use of kubectl application, which carries out HTTP requests to the Kubernetes API. |
|
sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 |
Exclude a legitimate IP address or range of addresses from the search. |
|
src_user=system:anonymous |
Search for anonymous users. |
|
|table src_ip src_user verb userAgent requestURI |
Display the results in a table with columns in the order shown. |
|
|stats count BY src_ip src_user verb userAgent requestURI |
Count the number of each unique combination of source IP address, user, user agent, and requests URI. |
|
|`kubernetes_aws_detect_suspicious_kubectl_calls_filter` |
Run this macro to speed up the search. |
Azure
- Ensure that you have configured Kube-Audit data diagnostics.
- Run the following search:
`kubernetes_azure` category=kube-audit
|spath input=properties.log
|spath input=responseObject.metadata.annotations.kubectl.kubernetes.io/last-applied-configuration
|search userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1
|table sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI
|rare sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI
|`kubernetes_azure_detect_suspicious_kubectl_calls_filter`
Search Explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
|
Splunk Search |
Explanation |
|
`kubernetes_azure` |
Run a macro to search Azure Kubernetes data. Note: The punctuation surrounding a Splunk macro is always a back tick (`), not a single quote ('). |
|
category=kube-audit |
Search the data source kube-audit from the diagnostic logs in Azure Cloud services. |
|
|spath input=properties.log |
Extract fields from the properties Kube-Audit log. |
|
|spath input=responseObject.metadata.annotations.kubectl.kubernetes.io/last-applied-configuration |
Return the field values of Kubectl calls. |
|
|search userAgent=kubectl* sourceIPs{}!=<valid IP address> sourceIPs{}!=::1 |
Find the wildcard string kubectl* to reveal the use of kubectl application, which carries out HTTP requests to the Kubernetes API. Exclude a legitimate source IP address or range of addresses from the search. |
|
|table sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace requestURI |
Display the results in a table with columns in the order shown. |
|
|rare sourceIPs{} verb userAgent user.groups{} objectRef.resource objectRef.namespace |
Display the least common source IP addresses, requests verbs, user agents, user groups, targeted resources, and namespaces. |
|
|`kubernetes_azure_detect_suspicious_kubectl_calls_filter` |
Run this macro to speed up the search. |
GCP
- Ensure that your deployment is ingesting Pub/Sub messaging logs.
- Run the following search:
`google_gcp_pubsub_message` data.protoPayload.requestMetadata.callerSuppliedUserAgent=kubectl* src_user=system:unsecured OR src_user=system:anonymous
|table src_ip src_user data.protoPayload.requestMetadata.callerSuppliedUserAgent data.protoPayload.authorizationInfo{}.granted object_path
|dedup src_ip src_user |`kubernetes_gcp_detect_suspicious_kubectl_calls_filter`
Search Explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
|
Splunk Search |
Explanation |
|
`google_gcp_pubsub_message` |
Run a macro to search GCP Pub/Sub Kubernetes data. Note: The punctuation surrounding a Splunk macro is always a back tick (`), not a single quote ('). |
|
data.protoPayload.requestMetadata.callerSuppliedUserAgent=kubectl* |
Find the string kubectl* to reveal the use of kubectl application, which carries out HTTP requests to the Kubernetes API. |
|
src_user=system:unsecured OR src_user=system:anonymous |
Search for unsecured or anonymous users. |
|
|table src_ip src_user data.protoPayload.requestMetadata.callerSuppliedUserAgent data.protoPayload.authorizationInfo{}.granted object_path |
Display the results in a table with columns in the order shown. Values from source IP addresses, source users, user agent, authorization granted information and the path of the object accessed via kubectl calls |
|
|dedup src_ip src_user |
Remove duplicate results from the same IPs and users. |
|
|`kubernetes_gcp_detect_suspicious_kubectl_calls_filter` |
Run this macro to speed up the search. |
Result
Kubectl is a tool that can do almost anything on a cluster, so it needs to be monitored. Unauthenticated calls indicate exposure of the API. Establishing security groups can limit API calls. Kubectl command strings can reveal malicious intent and likely access key compromise. Look at data such as geolocation, unusual users, unusual commands, request verbs, and object path.
Comments
0 comments
Please sign in to leave a comment.