Comprehensive coverage and cost-savings with Microsoft Sentinel’s new data tier

As digital environments grow across platforms and clouds, organizations face the dual challenge of collecting relevant security data to improve protection and optimizing the cost of that data to meet budget constraints. Management complexity also becomes an issue as security teams use diverse data sets to support on-demand investigations, proactive threat hunting, ad hoc queries, and long-term storage for audit and compliance purposes. Each log type requires a specific data management strategy to support its use cases. To address these business requirements, customers need a flexible security information and event management (SIEM) with multiple data layers.

Microsoft is excited to announce the public preview of a new data layer. Auxiliary log and Summary Rules In Microsoft Sentinel Expand your security coverage for large volumes of data at an affordable price..

Auxiliary Logs supports high-volume data sources, including network, proxy, and firewall logs. Customers can preview Auxiliary Logs today for free. We will notify users in advance before billing begins at $0.15 per Gb (East US). Initially, Auxiliary Logs allows long-term storage, but on-demand analytics are limited to the last 30 days. Additionally, queries are limited to a single table. Customers can continue to build custom solutions using Azure Data Explorer, but Auxiliary Logs is intended to cover most use cases over time and include management capabilities built into Microsoft Sentinel.

Summary rules further enhance the value of secondary logs. Summary rules allow customers to easily aggregate data from secondary logs into summaries and route them to analytics logs, giving them access to the full set of Microsoft Sentinel query capabilities. Combining secondary logs with summary rules enables security capabilities such as indicators of compromise (IOC) lookups, anomaly detection, and monitoring of unusual traffic patterns. Using secondary logs and summary rules together provides customers with greater data flexibility, cost efficiency, and comprehensive coverage.

The main advantages of auxiliary logs and summary rules are:

• Cost-effective coverage: Secondary logs are ideal for collecting large amounts of detailed logs at a low cost. When advanced security investigations or threat hunting are required, summary rules can aggregate secondary log data and route it to the analytics log tier, providing additional cost savings and security value.

• On-demand analytics: Auxiliary Logs support 30 days of interactive queries using limited KQL, facilitating access and analysis of critical security data for threat investigation.

• Flexible storage and preservation: Auxiliary records may be kept for up to 12 years for long-term storage. Access to these logs is possible via: Run a search task.

Multi-tier data collection and storage options in Microsoft Sentinel

Microsoft is committed to providing customers with cost-effective and flexible options to manage data at scale. Customers can choose from a variety of log plans in Microsoft Sentinel to meet their business needs. Data can be collected as analytics, primary, and secondary logs. It is important to distinguish what data to collect and where to collect it. We recommend categorizing security logs into primary and secondary data.

• Primary log (analysis log): Contains data with critical security value and leverages it for real-time monitoring, alerting, and analytics. Examples include Endpoint Detection and Response (EDR) logs, authentication logs, audit trails from cloud platforms, Data Loss Prevention (DLP) logs, and threat intelligence.
  • Typically, basic logs are monitored proactively, with scheduled alerts and analysis enabling effective security detection.
  • In Microsoft Sentinel, these logs are sent to the Analytics Logs table, allowing you to leverage the full value of Microsoft Sentinel.
  • Analysis logs can be retained for 90 days to 2 years, with a longer-term retention option available for up to 12 years.

• Secondary log (auxiliary log): Long, low-value logs that have limited security value but can help paint a full picture of a security incident or breach. They are not often used for in-depth analysis or alerting and are often accessed on-demand for ad hoc queries, investigations, and searches.
  • This includes NetFlow, firewall, and proxy logs and should be routed to either the primary or secondary logs.
    • Auxiliary logs are ideal for transforming data using Log Stash, Cribl, or similar tools.
    • If you don’t have a conversion tool, we recommend using the default log.
  • Both primary and secondary logs are available for 30 days, with a long-term retention option for up to 12 years.
  • Also, customers with extensive ML, complex hunting workloads, and frequent and extensive long-term retention may choose ADX. However, this adds additional complexity and maintenance overhead.

Microsoft Sentinel’s native data tiering gives customers the flexibility to collect, store, and analyze all their security data, helping them meet their growing business needs.

Example Use Case: Applying Auxiliary Logs and Summary Rules to Firewall Logs

Firewall event logs are a valuable source of network logs for threat hunting and investigation. These logs can show unusually large file transfers, the volume and frequency of host communication, and port scanning. Firewall logs are also a useful source of data for a variety of non-traditional hunting techniques, such as stacking transient ports or grouping and clustering different communication patterns.

In this scenario, organizations can now easily send all their firewall logs to Auxiliary Logs at a low cost. Customers can also create scheduled aggregations and run summary rules that route them to the Analytics Logs layer. Analysts can use these aggregations for their daily work, and when drill-down is required, they can easily query Auxiliary Logs for relevant records. Using Auxiliary Logs and summary rules together, security teams can use massive amounts of detailed logs to meet security requirements while minimizing costs.

Figure 1: Collects large amounts of detailed firewall logs into auxiliary log tables.

Figure 2: Creates an aggregated data set for detailed logs of auxiliary log plans.

Customers are already finding value in auxiliary logs and summary rules, as you can see below.

“The BlueVoyant team was excited to participate in the private preview of Auxiliary Logs and appreciate how Microsoft is creating new ways to optimize log collection with Auxiliary Logs. These new capabilities will allow us to transform previously undervalued data into more insightful and searchable data.”

Mona Gardiri

Senior Director of Product Management, BlueVoyant

“The Auxiliary Log provides the best functionality with the perfect fusion of the Basic Log and long-term preservation.
It’s the best of both worlds. When combined with summarization rules, it effectively addresses a variety of use cases for ingesting large volumes of logs into Microsoft Sentinel.”

Devak Manikandan

Senior Cyber ​​Security Engineer, DEFEND

expect

Microsoft is committed to expanding the scenarios covered by Auxiliary Logs over time, including data transformation and standard tables, large-scale query performance improvements, billing, and more. We are working closely with customers to gather feedback and will continue to add more features. As always, we would love to hear your thoughts.

Learn more