Leveraging Azure native tooling to hunt Kubernetes security issues

Container binary drift is the phenomenon in which a running container diverges from its original image over time. This can happen for a variety of reasons, such as manual updates, automated processes, or security vulnerabilities. Essentially, containers begin to diverge from the static snapshots they were created from, leading to potential inconsistencies and security risks.

When thinking about container image drift, it is important to understand the following:

• Security Risk: Image drift can pose a security risk because containers can run software or processes that were not included in the original image. This can create a security blind spot because traditional image scanning may not detect these changes.
• detection: Image drift detection involves monitoring containers for changes that differ from their original image. This can be done using tools that compare the state of a running container to its original image.
• prevention: To prevent image drift, we recommend implementing image invariance, regularly updating base images, and using image scanning tools. Monitoring and alerting on image drift can also help identify and address deviations.

In this three-part series, we’ll explore:

Part 1: How to Extend Your Capabilities Using the Latest Detection “Binary Drift” and Microsoft XDR Portal https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender-portal. We’ll also look at what the native integration between Defender for Cloud and Microsoft XDR can do for you, and why it’s beneficial for your SOC team.

Part 2: We’ll further extend the integration capabilities and show you how to automate your hunting using custom detection rules. https://learn.microsoft.com/en-us/defender-xdr/custom-Detection-rules. Reduce operational burden and proactively detect Kubernetes security issues. Where applicable, we also suggest alternative methods for performing detection.

Part 3: We’ll show you how you can leverage Defender for Cloud and Security Copilot in the XDR portal for Kubernetes security use cases leveraging AI.

memo: To limit the discussion, we assume that the container workloads are running on Azure Kubernetes Services (AKS) and that the AKS cluster leverages Azure’s RBAC.https://learn.microsoft.com/en-us/azure/aks/azure-ad-rbac) We also assume that you are using Azure Container Registry (ACR) to store your images.https://learn.microsoft.com/en-us/azure/container-registry/container-registry-concepts)

Let’s discuss what you need to set up in your environment to detect and classify drift. (Not all drift is malicious, it’s very likely that it’s a user or pipeline error.)

Setting up Defender for containers

We assume you have already enabled Defender for Containers. If not, follow the instructions listed here. https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-enable?tabs=aks-d…

Detect by setting up Defender for containers. Binary drift. This feature is available on Azure (AKS), Amazon (EKS), and Google (GKE) clouds.

To detect drift, you need to set: Drift Policy (https://learn.microsoft.com/en-us/azure/defender-for-cloud/binary-drift-Detection#configure-drift-po…) These policies define what you want to be notified about and what you don’t. You can create exclusions by setting higher priority rules for specific scopes or clusters, images, pods, Kubernetes labels, or namespaces.

In the sample rules below:

Figure. Binary drift detection rules

1. Scope Description: A human-understandable description of where we want to detect binary drift.
2. Cloud coverage: Refers to Azure, AWS, or GCP to which the rule applies. Expanding a cloud provider allows you to select specific subscriptions. If you do not select the entire cloud provider, new subscriptions added to the cloud provider will not be included in the rule.
3. Resource Scope: Here you can narrow the scope to specific objects (container name, image name, namespace, pod label, pod name, or cluster name).
4. Allow process list: List of processes that do not trigger alerts at a specified time Resource Scope

Also, each rule has a priority and is evaluated in ascending order. There is a default rule that ignores binary drift detection.

Figure. Basic rules

Prerequisites for creating notifications

Once you set up rules, they will be deployed to your Kubernetes nodes using the enhanced sensors in Defender for Containers. https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-introduction#sens…. You can check Defender for container settings. https://portal.azure.com/#view/Microsoft_Azure_Security/DataCollectionBladeV2/subscriptionId//defendersPlan/container

Figure. Defender Sensor must be enabled.

Once you have set up binary drift detection rules and enabled the Defender Sensor, you are ready to detect binaries that are not built from the original image but are still running.

Notification Review (Case Study)

In the “Security Warning” window, you will see a warning like this:

Figure. Binary drift warning

For example, the image the container is running on does not contain: Get

The attacker probably got his hands on this container and downloaded this utility to get some tools.

Notifications provide information about where the activity occurred, including object namespaces, images, clusters, etc.

This information may or may not be enough for you to take action. For example, if you want to identify “how” this drift occurred, that the user logged into the container and downloaded the binary. To supplement the information provided in the notification, you can use: Defender XDR Portal (https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender-portal)

In this article, we showed you how to leverage binary drift detection, and in the next article, we will focus on how to use the XDR Portal to build more context around these alerts and perform hunting.

I’ll also share some questions you can use as a starting point. [Part 2 ].