Using Defender XDR Portal to hunt for Kubernetes security issues

As we have seen Previous articleBinary drift alerts provide information about where activity occurred, including object namespaces, images, and clusters.

This information may or may not be enough for you to take action. For example, if you want to identify “how” this drift occurred, whether the user logged into the container and downloaded the binary. To supplement the information provided in the notification, you can use: Defender XDR Portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender-portal)

Harness the power of the Microsoft ecosystem

If you are an E5 customer, your security team is probably very familiar with: pre-hunt In the security.microsoft.com portal. Here we extend that hunting capability to add context to Kubernetes alerts. This is a huge time-saver and cost-effective way to train your Red Team or SOC analysts on level 400 Kubernetes concepts. To get started with Kubernetes hunting, you can leverage the developer knowledge of your platform team to provide the most common Kubernetes operations, such as exec (accessing containers) and debug (accessing nodes). Using KQL, your hunting team can leverage this in their hunting queries in the data structures and formats they already know.

Enhance your hunting with the XDR Portal

Defender for Cloud forwards incidents and alerts to the Defender Portal.https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-security-center-defender-cloud#investig…).

Similarly, Microsoft Sentinel also sends data to the Defender Portal.https://learn.microsoft.com/en-us/azure/sentinel/microsoft-365-defender-sentinel-integration?toc=%2F…)

Since your SOC and Red Team are already comfortable using the XDR Portal, Kubernetes hunt can now be easily incorporated into their playbooks.

When you look at alerts and incidents in the XDR Portal, you get a bird’s eye view of what, who, and where. This helps you further narrow your searches in Hunting Queries.

Figure. Attack story on Binary Drift

The Evidence and Response tab displays all relevant evidence. You will probably use these fields to create your hunting query.

Figure. Kubernetes objects related to the incident.

This integration allows you to run Advance Hunting queries using: Cloud Audit Event Table with Defender for Cloud Data.

The query below finds: Executive In a named pod Ubuntu (Here we assume that this pod is also running a container. Ubuntu Where the drift occurred and where the warning was generated)

CloudAuditEvents
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and RawEventData.ResponseStatus.code == 101  
| where RawEventData.ObjectRef.subresource == "exec"
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName startswith "ubuntu"
| extend Commands =  extract_all(@"command=([^\&]*)", RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, " "))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace,  Username, ParsedCommand

The above query produces the following, where you can see who ran the command, when it was run, and several other fields that help with context.

Figure. Query output

Another scenario to be aware of is activities performed on managed identities that may be related to notifications.

CloudAuditEvents
| where RawEventData.principaloid == 

This query provides the operations performed by this managed identity. This could be enumerating key vaults, storage accounts, etc. As before, you can examine the output such as timestamp, geo, etc. to determine if this operation is malicious.

Then, we can see the activities performed during this period as follows:

CloudAuditEvents
| where AzureResourceId == 
| where TimeGenerated > datetime() and TimeGenerated < datetime()
| where OperationName == "create"
| where UserAgent has “kubectl”

If the attacker performed an “exec” during this period, you can see it below: “ObjectRef” and “Request URI” Shows the exact command that was executed.

Note that query results will be displayed in your local time zone, depending on your settings. However, Kusto filters work in UTC.

Process automation and extension of basic detection capabilities

Now that we have defined a query that provides additional context for drift, we can turn it into a custom detection rule that can be run at a defined frequency.

For example, let’s say you want to be notified about pods that have permissions.

These pods run with a high set of privileges required to perform their tasks, but can also be used as a starting point to gain higher privileges.

Note: You can also prevent this by using Azure Policy built-in definitions.

https://portal.azure.com/#blade/Microsoft_Azure_Policy/PolicyDetailBlade/definitionId/%2Fproviders%2…

CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and isnull(RawEventData.ObjectRef.subresource)
| where RawEventData.ResponseStatus.code startswith "20"
| extend PodName = RawEventData.RequestObject.metadata.name
| extend PodNamespace = RawEventData.ObjectRef.namespace
| mv-expand Container = RawEventData.RequestObject.spec.containers
| extend ContainerName = Container.name
| where Container.securityContext.privileged == "true"
| extend Username = RawEventData.User.username
| extend DeviceId = RawEventData.AzureResourceId
| project Timestamp, ReportId, DeviceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, ContainerName, Username

Note that we will be using “Timestamp, Report ID, Device ID Turns a query into a custom detection rule, which can then become a starting pattern for a rule.

To create a custom detection/rule, you must include the columns mentioned below. https://learn.microsoft.com/en-us/defender-xdr/custom-Detection-rules#required-columns-in-the-query-…

Now you can easily convert this query into a custom detection rule.

Figure. Creating a custom detection rule

Complete the fields that will appear in the notification

Figure. Permissioned Pod Notification Details

In our case, we select the affected object, DeviceId (aka AzureResourceId).

Figure. IDs affected by the warning

Select an action

Figure. Actions to take when a warning occurs

Finally, create a notification.

Figure. Notification created

Once a rule is created, it will appear in the Custom Detection Rules section of the XDR portal.

Figure. Custom detection rules

As you’ll see in this two-part series, unlike many third-party solutions, it leverages your existing investment in the Microsoft security ecosystem, allowing you to conduct deep hunting using tools you already know.

Our proposal is as follows:

1. Work with your platform engineering team to identify the most risky Kubernetes operations relevant to your environment. For example, if you use: Destroyers images exec won’t make much sense
2. Provide SOCs and Red Teams with XDR Portal and Defender for Cloud integration documentation to leverage starting queries and customize them to their environment.
3. Review third-party Kubernetes security solutions to determine if the value of features not available in Native is worth the investment. In discussions with dozens of customers, Native has proven to be the most straightforward, cost-effective, and easiest to use and manage in the long term.
4. Additionally, if you haven’t considered Native-First in your security strategy, reevaluate your choices and consult with your security vendor about their latest roadmap for Native Security Services.
5. Remember, many on-premise security challenges have been due to a reliance on “best of breed” solutions. This strategy doesn’t work well in the cloud. Share these Native First security resources with your decision makers.