Answer key for trace; MemoryUsage1.etl

[itoleck]

Scenario:

A Windows Server is showing memory usage fluctuations in Task Manager.

Analysis:

The first thing to check when dealing with memory usage situations and something I like to check in all traces where the overall memory usage is captured is to look at the amount of RAM .vs the overall memory commit usage. If the commit usage is more than RAM, then the machine is utilizing the paging file, which is slower than RAM. If this condition persists, the RAM should be upgraded on the machine if possible.

Open the Memory panel on the Graph Explorer and open a new Memory Utilization analysis window.

Since the Total Physical Memory (RAM) is larger than the Commit (Total), the server's overall memory usage is within an acceptable amount. If the Total Physical Memory (RAM) was less than the Commit (Total) that would indicate that the server needs more RAM installed for best performance.

There are some other memory related graphs available in the Graph Explorer. Add the VirtualAlloc Commit LifeTimes graph to an analysis window.

In the analysis window, sort by the Size (MB) column if it is not already sorted. The carat above the column name indicates that the column is sorted. View the processes responsible for the memory usage. The memory usage is also graphed, and you should see the memory spiking throughout the trace period.

The processes responsible for the memory spikes are powershell.exe. The process IDs (PID) are different for each, so these are all separate processes. It may help to know what command was issued when the powershell.exe was called so that we know if there was any script ran in the command arguments. This information is available in the System Activity panel in the Graph Explorer.

Open the Processes graph in an analysis window. The column grouping by default is by Lifetime, then Process. The Lifetime column indicates when the process ran. If it is Transient that means that the process started and stopped within the trace period. A permanent process lifetime indicates that the process started before the trace period. All of the powershell.exe processes that have memory usage spikes are transient.

A good analysis view is shown below; with the memory graphs and process graphs in the same analysis tab, filtered and sorted by the powershell.exe process. When you select each powershell.exe process, you will also select the memory spike that correlates to the process.

In the VirtualAlloc Commit LifeTimes graph there are also columns available for Type and Impacting Stack (Frame Tags). The Type will show that the memory allocations were Allocated Inside Freed Inside (AIFI), this means that the memory was used and cleaned up during the trace period. This is usually indicative of normal memory usage. Look for processes that are (AOFO) or (AIFO) when looking for memory leaks.

The Impacting Stack (Frame Tags) column is n/a which indicates that no stack information for the memory allocations is available. This can be verified by looking in the System Configuration -> Trace Statistics window. In this case there are StackWalk events captured in the trace. There are other reasons that the stack may not be available, in this case it is because the trace is missing the ImageId Provider trace events. A new trace would have to be taken with the correct providers to understand more about the functions called in the powershell.exe processes.

Below is a good example of the trace events with ImageId and StackTrace events.

Remediation:

No remediation needed, but if the PowerShell.exe processes that capture performance information become an issue; stop the collections.