How submissions help improve Defender for Office 365

Microsoft Defender for Office 365 allows users and administrators to submit suspicious items (emails and Teams messages, files, or URLs) for analysis to enhance detection and prevention. Submissions help Microsoft understand the nature of the items, update filtering decisions, and provide actionable insights. insight. There are frequently asked questions about what happens after you submit an item to Microsoft, so here’s a brief explanation of what happens behind the scenes.

You can submit items to Microsoft Defender for Office 365 in a variety of ways, depending on your role and the source of the item. For example, you can submit items from:

• field of visionlater Configure user reporting settings And the reported message destination Microsoft only or Microsoft and Reporting Mailbox.
• Microsoft Defender XDR Submission, Quarantine page or Take Action wizard (for organizations with Microsoft 365 E5 or Microsoft Defender for Office 365 Plan 2 licenses) or Enable real-time detections (for organizations with Defender for Office 365 Plan 1 licenses). You can immediately resolve false positives and false negatives by creating Allow or Block items while submitting items.
• Tracking messages In the Exchange Administration Center (EAC)

tip: From a feedback perspective, there is no difference between a user-reported item and an admin-submitted item. It’s just how different personas report items.

We recommend that you configure your message target to one of the following: Microsoft only or Microsoft and Reporting Mailbox ~ in User-reported settings. This configuration again ensures that the administrator does not have to resubmit the user report. If the user reporting settings are configured to send messages only to the reporting mailbox, the security team must actively submit the user report via the administrator submission.

When a user or administrator submits an item to Defender for Office 365, the following steps occur:

Phishing simulation check

• Make sure the email is your primary email. Attack simulation training Or configured through third-party phishing simulation training Advanced delivery policy for third-party phishing simulations. If so, you will see “Phishing Simulation” in the User Reports view on the submission page. result heat.
• User-reported “Phishing Simulation” type items do not generate alerts and no Automated Investigation and Response (AIR) investigations are performed.
• This item is no longer being analyzed.

Verify authentication

Policy Hit

• Microsoft Defender for Office 365 filtering decisions have been overridden by an administrator override (Exchange mail flow rules (transport rules), anti-spam policy allow/block, anti-spam policy settings, or tenant allow/block lists).
• User overrides (e.g., user-defined Outlook settings such as trusting email from contacts, user safe senders list (including domains), user blocked senders list (including domains), etc.) override Microsoft Defender for Office 365 filtering decisions.
• Microsoft Defender Filter for Office 365 detected the message, but the action was blocked due to administrator configuration (e.g. Preventing Impersonation in Anti-Phishing Policy (is turned off).

Defender for Office 365 filters check for the latest verdict (revalidation).

Sometimes Microsoft Defender for Office 365 has already caught up with the Indicators of Compromise (IOCs; URL/Attachment/Sender/IP) associated with the submission. This updated decision may be due to changes in the reputation of the sender/IP, or detection of URLs and files associated with delayed weaponization (initially clean, but later malicious).

• These items are then re-scanned by various engines and systems (e.g. antivirus, anti-spam, machine learning).
• If the retest results are updated, the submission will display the latest results.
• The Microsoft Defender for Office 365 filter has started classifying the item correctly, so it is no longer being analyzed.

rating

Submitted items are analyzed and classified by a mix of state-of-the-art automation and human evaluators. Evaluators review messages, URLs and attachments, QR codes, and all metadata associated with the submitted item.

Based on the combination of these 5 steps, you can get results, result details and recommended steps. You can find the full set of results. here.

In US government agencies (Microsoft 365 GCC, GCC High, and DoD), administrators can submit items to Microsoft for analysis, but items are only analyzed for phishing simulations, authentication, and policy hits. Rescanning and scoring analysis is not performed for compliance reasons (data is not allowed to leave organizational boundaries).

Why does Microsoft verify submissions? Why does Microsoft sometimes disagree with submissions?

While we know that most customers take precautions before submitting items to Microsoft, trusting every submission can be a major security vulnerability. Some email evaluations can be subjective (what is spam to one user may be acceptable to another). Some submissions may contain human error. Therefore, it is important to verify your submission before Microsoft uses it to update its filters.

If Microsoft’s final rating differs from what was reported, we ensure that we have a strong signal of disagreement. For example, if an email is submitted as clean, but a human analyst finds compelling evidence of malicious entity/intent, the submission results will reflect the human analyst’s verdict. Microsoft strives to reduce instances of false disagreements. However, there may be extremely rare cases (human error) where a customer may feel that a disagreement is invalid.

Why does Microsoft sometimes produce results as Unknown?

This ruling was made for two reasons.

1. Despite our rigorous rating process, sometimes we cannot make any conclusions about the classification. For example, by the time we grade, the URL in the email may not be accessible. We continuously monitor these samples to reduce unknowns and research and design techniques to classify relevant items in the future.
2. Given the volume of submissions received by Microsoft, human analysts cannot review them in a timely manner. We monitor our analytics coverage on a daily basis and invest in ensuring that each submission is reviewed in a timely manner.

If a submission is identified as malicious (false negative), Microsoft takes one or more of the following actions:

• Runs a system-wide zero-hour auto-purge (ZAP) operation on messages containing IOCs (URLs, files, fingerprints, etc.). ZAP has the following requirements: Zero Hour Automatic Cleanup (ZAP) for Microsoft Defender for Office 365.
• Add specific IOCs to the system-wide Microsoft Defender for Office 365 internal block list.
• Retrain machine learning models and other heuristics using labels as input.
• Add manual rules that target specific patterns missed by Microsoft Defender for Office 365.
• Invest in sustainable improvements to attack methods that require the use of innovative technologies (e.g. QR code based attacks).

If a submission is identified as a false positive, Microsoft takes one or more of the following actions:

• Microsoft Defender for Office 365 marks certain IOCs as safe.
• Retrain machine learning models and other heuristics using labels as input.
• Updates existing manual rules that target specific patterns captured by Microsoft Defender for Office 365.
• Invest in innovative technologies and techniques to identify less legitimate emails as malicious.

For user-reported phishing messages, Automated Investigation and Response (AIR) Triggered by an alert. AIR clusters all related messages and then analyzes them to determine if the original email was malicious. If a user-submitted cluster is deemed malicious, it recommends remediation actions for the entire cluster to the SOC team, increasing the effectiveness of the SOC team in remediating and responding to threats. Automated feedback to end users.

Even if your submission is accepted as valid, our technology is designed for long-term improvement and the steps to make sustainable changes to the Microsoft Defender for Office 365 filtering stack (bulk/phishing/spam/malware/clean) are not always simple. Therefore, there are a number of reasons that may impact the immediate nature of the change, including:

• Taking the simple approach of adding a sender or IP to a global allow/block list can have devastating effects if done wrong. This carries a high risk, especially at the scale that Microsoft operates (some email filtering solutions on the market can take this risk). For example:
  • A legitimate domain that has been hacked and used to send malicious campaigns. Marking a domain as bad for a long period of time can have significant repercussions for the legitimate domain owner.
  • There may be domains owned by attackers that initially build up a good reputation and submit emails as clean. They can use this as a “free pass” to send out threat campaigns, which can have devastating effects on numerous Microsoft customers.
  • Attackers often use shared infrastructure that can be used by both legitimate senders and malicious actors.
• Machine learning (ML) models are trained on vast amounts of data. Often, a single submission from a single customer is not enough to change behavior. This is not an excuse, but it helps to highlight the trade-offs that these ML models make in exchange for explainability.
• Email judgment (spam, bulk) can sometimes be subjective. What is spam to one user may be acceptable or desirable to another. Outlook end-user allow/block allows for personalization.
• Given the size of the submissions, manual rules written by security researchers only apply to submissions that are part of larger clusters (email messages containing the same IOC).

For all the reasons explained above, we are making several ongoing investments to improve the process of learning from submissions, but some customers may not see immediate changes after submission. We recommend using the tenant allow/block list action available during submission ( Perform Task Wizard or submit Take immediate remedial action via the Defender for Office 365 page and let Defender for Office 365 manage expiration based on learning time.

Submissions are the most important source of information that helps us improve Defender for Office 365. We continue to encourage you to submit submissions to resolve FP/FN issues and only use support tickets when you feel additional intervention is needed. Defender for Office 365 leverages all submissions, even if they may not detect immediate changes. In addition to contributing to collective security intelligence by submitting items as spam, phishing, malware, or clean, you also indirectly help Defender for Office 365 improve and combat attackers. In turn, submissions help your organization and other customers across the service by reducing the number of unwanted email messages and preventing legitimate email messages from being flagged inadvertently.

More information