Troubleshoot Policy Issues for Sending logs to Log Analytics/Storage Account for Azure Resources

Azure Custom Policies

Azure custom policies are used to extend and customize the behavior of Azure services to meet specific requirements that are not addressed by built-in policies.

Reference on how to troubleshoot common issues while creating custom policies can be found here. Troubleshooting common custom policy issues in policy development

Custom policies for sending logs to log analysis workspaces and storage accounts

Use custom policies to send logs to a Log Analytics workspace and a storage account in Azure to ensure that all required logs are collected consistently and automatically and stored for analytics, audit, and compliance purposes. By using custom policies to send logs to a Log Analytics workspace and a storage account, organizations can enhance their logging and monitoring capabilities, improve security and compliance, and streamline their log management processes.

How to troubleshoot common issues with custom policies for sending logs to log analysis workspaces and storage accounts

While adding custom policies and assigning them to specific subscriptions, you may encounter several issues in your compliance reports, which can sometimes be difficult to understand and fix. Below is a list of use cases and fixes that will help you quickly identify the root cause and resolve the issue.

Use Case 1 – Logs are enabled but resources are still marked as noncompliant

Below is an example of a function app with logs enabled, but resources still appear as noncompliant.

fix: Check your diagnostic settings and make sure all logs are enabled as per the policy rules mentioned in the policy definition. If all logs are not enabled, the policy rules will not match the settings and the resource will be marked as noncompliant.

Now you can see that all the logs are not enabled as shown in the screenshot of the feature app below.

To resolve this issue, go back to your policy definition and add entries for all your logs in the “Logs” section. Log categories vary by product. Always check your log categories before adding a log policy.

Case 2 – Incorrect noncompliance reason for a resource that does not have logs enabled.

Sometimes resources are non-compliant, but sometimes the reasons for non-compliance are incorrect.

Below is a screenshot of the reason for noncompliance. This is a very common issue observed across multiple products.

The current value is displayed as follows: Both “true” and “false”.

fix: To resolve this issue, check if your product has multiple log categories and follow the steps below.

1. If the resource has multiple log categories

If your product has multiple log categories, you should add a “count” variable to your policy definition. This will allow you to check all the logs, count the value, and mark the resource as compliant or non-compliant based on that.

For example, for a batch account, as shown in the screenshot below, there are two different log categories greater than 1. Here, we need to put a count variable so that we can calculate the total logs and reflect the correct compliance reports.

Here in the policy definition we have put a count variable for “All logsThis is only possible in “Category Groups” since there are no multiple categories for “All Metrics”. If there are multiple categories for “All Metrics”, you can also put a count variable for “All Metrics”.

Please leave a note to check if the “All Logs” category is part of a “Category Group”. You can always check this by checking the JSON view for your diagnostic settings.

2. If the resource has all logs and audit logs

All log categories and audit log categories are different for each product. For some products, you will notice that all logs also include audit logs. That is, when you enable all logs, audit logs are automatically enabled, but for some products, all logs and audit log categories are completely different. That is, you need to explicitly enable all logs and audit logs and add the same logic to your policy definition. If you do not do so, the compliance report will mark the resource as not in compliance with the following reason: [true, false]True False means fewer active logs, and fewer active logs.

As per the screenshot below, the logs are enabled but the audit logs are not enabled. This is a scenario where the compliance report shows the resource as noncompliant because the logs are partially enabled, and therefore the noncompliant reason becomes True and False.

Also, if logs are not enabled, the resource will be marked as noncompliant. This is because the policy definition only checks all logs and not audit logs, which is why true and false are the same.

fix: To resolve this issue, double-check the JSON view of your diagnostic settings and ensure that there are separate category groups defined for Audit and All logs as shown in the screenshot below, then update your policy definition to include Audit logs as well.

The updated policy definition is as follows:

Updating the policy definition will result in compliance reports showing the correct reasons for noncompliant resources that do not have logs enabled. Compliance reasons include:

Additionally, for resources with partially enabled logs, running a remediation action will fully enable the logs and mark the resource as compliant.

Case 3 – No multiple log categories and noncompliance reasons are misleading

If a noncompliant resource is displayed for the following reasons: “True, false” If the above mentioned use case does not fit your criteria, add the same policy to another subscription and check the compliance reason. If the non-compliance reason changes, you should check the HAR trace log to check the values ​​for get and post requests and make a decision.

This may be a UI issue, so you should contact the PG team to figure out a fix for it.

For example, see the screenshot below for a log policy, where the noncompliance reason changes in two different subscriptions, but the policy for sending logs to log analytics remains the same in both subscriptions.

In the first screenshot, the reason for failure is blank, but in the second screenshot for the same policy, the reason for failure is blank, but it says “No related resources matching the effect details”, which is misleading. This is a UI issue, the policy logic is correct here.