Container, Security, Kubernetes

introduction

In cloud-based Kubernetes environments, containers are often treated as follows: Unchangeable Since they are resources, they should not be changed after deployment. Immutable containers minimize the attack surface because they do not allow modifications during runtime. This limits the possibility of an attacker making unauthorized changes, installing malware, or creating backdoors within a running container.

Container drift refers to unintended or unauthorized manual changes, updates, patches, or other modifications made during runtime. When a container drifts, it can incorporate untested and unvalidated changes, such as software updates, configuration modifications, or new libraries. These changes can introduce new vulnerabilities that were not present in the original validated container image. Drift can also introduce changes that grant elevated privileges to processes or users within the container, which can be exploited to gain broader access to the system or network. Changes caused by drift can alter or disable security monitoring tools within the container, making it difficult to quickly detect and respond to security incidents.

Microsoft Defender for Containers introduces binary drift detection in public preview to detect file executions in running containers that drift from the original container image that has been scanned, tested, and validated. Azure (AKS) Version 1.29Amazon (EKS) and Google (GKE) cloud.

Defender Binary Drift Detection for Containers helps organizations:

Early detection of breaches: Drift detection acts as an early warning system for potential security breaches. If an attacker compromises a container and makes unauthorized changes, drift detection immediately alerts the security team so they can respond quickly and mitigate the impact.

Monitoring Insider Activity: Drift detection helps mitigate insider threats by monitoring for unauthorized changes that may indicate malicious insider activity. This includes unauthorized changes to configuration, deployment scripts, or access controls within containers.

Reduce human error: Human error is a common cause of security breaches. Drift detection reduces the risk of human error by quickly detecting and correcting unintended changes made by administrators or developers.

Ensuring compliance with security standards: Many regulatory standards require organizations to maintain secure configurations and prevent unauthorized changes. Drift detection helps ensure compliance by continuously monitoring and documenting container status, providing evidence that configurations are consistent with regulatory requirements.

Prerequisites for enabling binary drift detection:

• Create and modify drift policies with at least Security Administrator privileges for the tenant.

Configuring binary drift detection

Security administrators can configure drift detection policies on Azure subscriptions, AWS connectors, or GCP connectors, and on resources at the cluster, namespace, pod, or individual container level.

For more information on how to configure drift detection rules, see: Binary Drift Detection (Preview) – Microsoft Defender for Cloud | Microsoft Learn

Rules are evaluated in ascending order of priority. The first rule 1 is evaluated, and if it matches, evaluation stops. If no rule matches, the next rule is evaluated. If no rule matches, the rule is executed immediately. Basic binary drift Rules with default values Ignore drift detection It applies.

Best practices for drift detection:

• Kubernetes administrators must ensure that all container images are regularly updated and patched to include the latest security fixes.
• Drift detection Cluster level Helps prevent unauthorized changes that could compromise the security and stability of your entire cluster. For example, an attacker with access to the Kubernetes API server could change cluster-wide settings to escalate privileges or disable security features.

• In multi-tenant environments where multiple teams or customers share the same Kubernetes cluster but operate within their own namespaces, organizations can apply drift detection at the namespace level to monitor only the areas of the cluster relevant to specific applications or teams.

• In a development or test environment, developers may need to make temporary changes to containers to test new features, configurations, or debug issues without the overhead of redeploying containers. A set of rules applies only to Kubernetes pods that are labeled with a specific label.

• During a scheduled maintenance window, organizations may need to apply emergency patches or make rapid operational changes directly to running containers to address critical security vulnerabilities or resolve urgent issues. In this scenario, you can modify rule actions to Ignore drift detection To avoid false positive results.
• Allowlist for processes – Organizations can define specific processes, such as monitoring agents and logging agents, to exclude from drift detection to avoid false positive results.

Binary Drift Warning Test/Simulation

To test the binary drift feature and generate alerts (only in cases where you want to be notified by the binary drift policy), you can run a binary process in a container (not part of the original image). You can also use this script to create binary drift scenarios.
kubectl run ubuntu-pod –image=ubuntu –restart=Never — /bin/bash -c “cp /bin/echo /bin/echod; /bin/echod This is running binary drift”

Below you can see the drift detection alert generated from the threat scenario.

Click Open Logs to see details of activities performed on this resource over the elapsed time of the alert. An attempt to list cluster administrator credentials was successful.

This warning also indicates that there are 42 more warnings on the affected resource.

This incident indicates that suspicious activity has been detected in your Kubernetes cluster. Multiple alerts across multiple Defender for Cloud plans were triggered on the same cluster, increasing the likelihood of malicious activity. Suspicious activity may indicate that a threat actor is attempting to gain unauthorized access to and compromise your environment.

Advanced Hunting with XDR

Security teams can now access Defender for Cloud alerts and incidents in the Microsoft Defender portal, and instantly correlate alerts and incidents to get a complete picture of attacks, including suspicious and malicious events occurring in their cloud environments.

By combining drift detection data with other security event information, SOC teams can build a more comprehensive understanding of potential incidents. Multi-stage incidents with multiple alerts can be observed in the XDR portal.

The Alert Evidence window shows that there was suspicious activity from “ubuntu-pod”.

The SOC team can use the queries below to further investigate the commands executed on the affected Pods and the users who executed the commands.

Cloud Audit Event

| If Timestamp > ago(1d)

| If DataSource == “Azure Kubernetes Service”

| If OperationName == “create”

| If RawEventData.ObjectRef.resource == “pods” and RawEventData.ResponseStatus.code == 101

| if RawEventData.ObjectRef.namespace == “kube-system”

| if RawEventData.ObjectRef.subresource == “exec”

| If RawEventData.ResponseStatus.code == 101

| extends RequestURI = tostring(RawEventData.RequestURI)

| extends PodName = tostring(RawEventData.ObjectRef.name).

| extends PodNamespace = tostring(RawEventData.ObjectRef.namespace).

| username extension = tostring(RawEventData.User.username)

| If PodName == “ubuntu-pod”

| Command expansion = extract_all(@”command=([^\&]*)”, requestURI)

| ParsedCommand = url_decode(strcat_array(command, ” “))

| Project Timestamp, AzureResourceId, OperationName, IPAddress, UserAgent, PodName, PodNamespace, Username, ParsedCommand

For more information on how to investigate suspicious Kubernetes (Kubeaudit) control plane activity with XDR Advanced Hunting, see: Kubeaudit Events for Advanced Hunting – Microsoft Defender for Cloud | Microsoft Learn

The SOC team can assign incidents. Incident Management A window to mitigate attacks

Kubernetes Cluster Manager You can configure automated workflows to handle common drift scenarios, such as reverting unapproved changes, notifying relevant teams, and more. Trigger response action Automatically.

Additional Materials

You can also learn more about these features using the following resources:

Reviewer

Eyal Gur, Senior Product Manager, Defender for Cloud