Troubleshooting High CPU Usage in IIS Worker Processes: A Comprehensive Guide

context:

High CPU usage by the IIS worker process (w3wp.exe) can significantly impact the performance of your web applications. This article describes tools you can use to identify symptoms, initial troubleshooting steps, and data collection methods.

Symptoms:

If the IIS worker process is using a lot of CPU, you may see the following symptoms:

• Your web application is experiencing slow response times or timeouts.
• High memory usage due to high CPU usage.
• Overall server performance has been degraded.

Initial Isolation:

Troubleshooting high CPU issues involves two main tasks: identifying the problem, collecting logs, and analyzing them. Here are some isolations that can help you start the IIS worker process:

1. Does high CPU usage mean high memory usage or slowdowns? Look at memory usage along with CPU usage.
2. Describe the steps that lead to high CPU usage. Does CPU consumption increase with load?
3. What is the CPU usage of w3wp.exe alone and what is the total CPU usage of the server while the issue occurs? If your overall CPU is high, there may be a server performance issue.
4. How does CPU consumption change over time? If this keeps spiking up and down, you should follow the code execution. If it monitors and persists for 5-10 seconds, there are additional troubleshooting scenarios.
5. How often does the problem occur and how high does the CPU go? Observe any patterns or specific times. This could be when scheduled tasks are running or load is increasing.

Data collection

For .NET Core applications hosted on IIS, the method of data collection depends on whether the app is in-process or out-of-process. In progress mode is recommended over out-of-proc.

Note 1: For CPU analysis ETW Tracking or Memory dump It can be useful. Don’t collect both at the same time to avoid additional CPU load or slowdown.

ETW Trace vs Memory Dump:

• ETW Tracking: Useful in production scenarios where performance impact is minimal. Captures specific events and performance counters.
• Memory dump: Provides an in-depth view of objects, values, and roots. Useful for detailed analysis.

Note 2: Do not collect logs at very high CPU values ​​(>=95%) as this may cause the server to hang or become unresponsive. Collect logs before CPU reaches that level.

The initial goal is to get a log that shows the work being done on the same non-waiting thread during some of the highest w3wp.exe CPU periods during the problematic period. So:

• Multiple dumps are required (usually 3 is sufficient).
• Dumps must be performed on the same process ID.
• Dumps should be close enough in time (usually 10 seconds apart).
• Collect dumps when CPU usage is high and considered abnormal.

ETW Steps – Perfview

Collect trace information using the Perfview tool.

1. Download Perfview here.
2. Run PerfView.exe with administrator privileges.
3. Problem occurs:

• Click the Collect menu and select a collection option.
• Check the Compress, Merge, and Thread Time checkboxes.
• Expand the Advanced Options tab and check the IIS checkbox.
• When a problem occurs, we start and stop collecting.

memo: Don’t capture this trace for too long, as the data will be overwritten once the circular MB limit is reached.

Memory dump steps

Use Procdump.exe or DebugDiag collection tool. Procdump is preferred when application installation is not possible.

Note 3: Do not use task manager Collect dumps. Specialized tools like DebugDiag and Procdump provide more detailed information and handle process bits correctly.

Option 1: Procdump

1. Download Procdump.exe here.
2. From a command prompt with administrator privileges, navigate to the directory where you extracted the downloaded procdump.zip and run the following command:
3. procdump.exe -c w3wpCPU consumption trigger -S Number of seconds -male Number of dumps to collect PD

• -c: w3wp.exe CPU consumption threshold. Change w3wpCPU consumption trigger The threshold is said to be 80%.
• -s: Number of consecutive seconds that CPU usage must exceed the threshold. Change Number of seconds The time in seconds that the CPU remains above a threshold (e.g. 10 seconds).
• -ma: Full memory dump.
• -n: Number of dumps to collect. Change Number of dumps to collect With the number of dumps to collect. This is usually set to 3 for deterministic analysis.
• PID is the process ID and can be found in Task Manager or IIS worker processes.

Option 2: DebugDiag

1. Install DebugDiag here.
2. Open DebugDiag Collection from the Start menu.
3. If necessary, change the dump path via Tools > Options and Settings > Save Manual User Dumps Folder.
4. Click the “Processes” tab.
5. Find the w3wp process by process ID.
6. Right click on the process and select “Create Userdump Series” and set the options.
7. Click “Save & Close” to start generating the dump.

finally: By following these steps, you can effectively isolate and collect the necessary data regarding the high CPU consumption issue of IIS worker processes. If you are comfortable with analyzing this data, you should be able to gain insight from Perfview an Dumps. If you would like to do so, please submit a case to Microsoft IIS support and we will take care of it for you.