Save ingestion costs by splitting logs into multiple tables and opting for the basic tier!

In this blog post, we will talk about how to split your logs into multiple tables in Microsoft Sentinel and choose the default tier to reduce costs. Before we get into the details, let’s understand what problem this approach solves.

Azure Monitor provides several log plans that customers can choose from based on their use cases. These log plans are:

• Analysis Log – This plan is designed for frequent, concurrent access and supports interactive usage by multiple users. This plan powers the capabilities of Azure Monitor Insights and powers Microsoft Sentinel. It is designed to manage critical and frequently accessed logs optimized for dashboards, alerts, and business-level queries.
• Basic Log – Improved to support richer troubleshooting and incident response with faster queries, while also reducing costs. Now with longer retention periods and KQL operators for aggregation and querying.
• Auxiliary log – A new, low-cost log plan that lets you collect and manage detailed logs required for audit and compliance scenarios. These can be sparsely queried with KQL and used to generate summaries.

The following diagram provides more details on the log plan and use cases.

You may also want to take a look at: Public Document Get detailed insights into feature-wise comparisons of log plans to help you make the right decision to choose the right log plan.

**Note** Auxiliary logs are beyond the scope of this blog post. I will write a separate blog about auxiliary logs in the future.

So far, we’ve looked at the different log plans available and their use cases.

The next question is which tables support Analytics and the default log plan.

You can switch between Analytics and Basic plans, and changes will be immediately reflected in existing data in your tables.

When you change a table’s plan from Analytics to Basic, Azure Monitor treats all data older than 30 days as long-term retention data based on the total retention period set for the table. That is, the total retention period for a table does not change unless explicitly changed. Modify the long-term storage period.

Check out ours Public Document For more information on how to plan your table, see here.

In this blog, we will focus on partitioning the Syslog table and setting up a DCR-based table as the default tier.

Typically, firewall logs contribute to the vast amount of log collection for SIEM solutions.

To manage costs in Microsoft Sentinel, we recommend that you thoroughly review your logs and identify which logs can be moved to the default log plan.

At a high level, the following steps will suffice to accomplish this task:

• Collect firewall logs to Microsoft Sentinel with the help of Linux Log Forwarder via Azure Monitor Agent.
• Assuming that the logs are collected in a Syslog table, create a user-defined table with the same schema as the Syslog table.
• Split the logs by updating the DCR template.
• Set table plan as default for identified DCR-based custom tables.
• Sets the required retention period for the table.

At this point, we expect to already have a log forwarder set up to collect firewall logs into a workspace in Microsoft Sentinel.

Now let’s focus on creating a custom table.

This part used to be a hassle, but not anymore, thanks to my colleague Marko Lauren who did a great job on this. PowerShell script Creating custom tables is easy. Simply enter the name of an existing table and the script will create a new DCR-based custom table with the same schema.

Let’s actually check it out.

• Download the script locally.
• Open the script in PowerShell ISE and update the Workspace ID and Resource ID details as shown below.

• Save it locally and upload it to Azure PowerShell.
• Load the file and enter the table name from which you want to copy the schema.
• Provide a new table name as you want and make sure the name has the suffix “_CL” as shown below.

This will create a new DCR-based custom table that you can view in the Log Analytics Workspace > Tables blade as shown below.

**memo** We strongly recommend that you review your PowerShell scripts thoroughly and perform appropriate testing before running them in production. We are not responsible for the scripts.

The next step is to update the data collection rule template to split the logs.

Since we have already created the custom table, we need to create transformation logic to split the logs and send the less relevant logs to the custom table which will be set as the default log tier.

For demo purposes, we will split the logs based on SeverityLevel. We will delete “info” logs from Syslog table and stream them to Syslog_CL table.

Let’s see how this works.

• Find the Data Collection Rules blade.
• Open the DCR of the Syslog table and click Export Template > Deploy > Edit Template as shown below.

• In the dataFlows section, we have created two streams to split the logs. Here are the details for the streams:
  • 1^castle Stream: Drops Syslog messages with SeverityLevel of “info” and sends logs to the Syslog table.
  • 2^~is Stream: Captures all Syslog messages with SeverityLevel of “info” and logs them to the Syslog_CL table.

Let’s verify that it really works

Go to Log Analytics Workspace > Logs and verify that the data you defined is in the table.

As you can see in my case, the Syslog table contains all logs except those with SeverityLevel of “info”.

Additionally, a user-defined table Syslog_CL contains Syslog data where SeverityLevel is “info”.

Now the next part is to set the Syslog_CL table as the default log plan.

Syslog_CL is a DCR-based user-defined table, so you can set it as your default log plan. The steps are simple:

• Go to Log Analytics Workspace > Tables
• Table search: Syslog_CL
• Click the ellipsis on the right as shown below and click Manage Tables.

• Select a table plan as your base and set your desired retention period.

Now you can enjoy the cost advantage. Hope it helps.