Skip to content

Add guidance for capturing ETW traces in Kubernetes pods - #2344

Merged
Brian Robbins (brianrob) merged 4 commits into
mainfrom
copilot/add-guidance-for-traces-in-k8s
Dec 3, 2025
Merged

Brian Robbins (brianrob) merged 4 commits into
mainfrom
copilot/add-guidance-for-traces-in-k8s

Conversation

Copilot AI commented Dec 1, 2025 •

Copy link
Copy Markdown
Contributor
  • Explore the repository structure and existing documentation
  • Understand the existing Windows containers section in UsersGuide.htm
  • Research EnableEventsInContainers and ImageIDsOnly options
  • Add new section "Capturing ETW Traces with Process-Isolation Windows Containers (Kubernetes)" to UsersGuide.htm
  • Address PR review feedback:
    • Clarify process-isolation is the default mode
    • Add note that Hyper-V mode doesn't require these instructions
    • Fix kernel session limitation explanation
    • Remove "(Required)" from Step 1 heading
    • Add that host user-mode events still work without /EnableEventsInContainers
    • Add note about building PerfViewCollect from source
    • Fix PDB signature terminology
    • Update error example to show "module!?" format
    • Clarify jitted code still works
    • Explain merge component container access limitation
  • Run code review
Original prompt

This section details on the original issue you should resolve

Guidance on How to Capture Traces in Kubernetes Pods
> Hi Brian Robbins (@brianrob). I'm currently running into a similar issue trying to run PerfView in a k8s pod. For those less knowledgeable with this tool is it possible to provide more context of what to do in this scenario?

Is the expectation on the Host node a user manually starts the PerfView process to collect a trace using the /EnbleEventsInContainer parameter. After they are done collecting the trace you then copy the zip into the container and perform a PerfView merge -ImageIDsOnly NameOfTrace.etl.zip? What does -ImageIDsOnly do exactly? I run that command and can't really tell if it worked.

If you don't run -ImageIDsOnly is it still possible to use that trace to profile what occurred in the k8s pod?

If this should be a new issue or discussion please let me know. Thanks!

Originally posted by C J (@Cjewett) in #1790

Please add a section to the PerfView users guide on how to capture ETW traces when one or more processes are in process-isolation Windows containers. It would be especially helpful if you provide example command lines, and describe what each command does, as well as which ones are required and which ones are optional. If they're optional, describe what happens if you don't run them.

Comments on the Issue (you are Copilot in this section)

Brian Robbins (@brianrob) There are some strong limitations in how we can best collect ETW traces on Windows when containers are involved. The happy path is the following:
  1. Capture an ETW trace on the host node. Specify /EnableEventsInContainers when collecting to ensure that user-mode events flow from processes inside of containers back to the ETW session. If you don't specify this, then you'll get all kernel events but only user-mode events from processes running directly on the host and not inside of a container.
  2. At this point, as long as the container(s) containing the process(es) of interest are still running, you can open and analyze the trace on the host node. PerfView will be able to find any binaries that it needs both on the host and inside the containers.
  3. If you want to shutdown a container of interest, or copy the trace to another machine for analysis where the container is inaccessible, then you'll need to copy the trace into the container, and then inside the container, run PerfViewCollect merge -ImageIDsOnly . Once it completes, copy the resulting file (overwrites the original) out of the container. This command is responsible for opening all of the accessible binaries that have symbols in the trace, and injects PDB information into the trace, so that we can download the PDBs from the symbol server later on during analysis. If you don't do this, then some symbols just won't be resolvable later. The data is still valid, but might be less useful.
    Brian Robbins (@brianrob)
    C J (@Cjewett), let me know if you have any questions on this. I'm also going to use this issue to get copilot to update the PerfView users guide with this information.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: brianrob <6210322+brianrob@users.noreply.github.com>
Copilot AI changed the title [WIP] Provide guidance on capturing traces in Kubernetes pods Add guidance for capturing ETW traces in Kubernetes pods Dec 1, 2025

@brianrob Brian Robbins (brianrob) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot, please address these comments.

Capturing ETW Traces with Process-Isolation Windows Containers (Kubernetes)

When running Windows containers in Kubernetes using process-isolation mode (as opposed to Hyper-V isolation),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarify that process-isolation mode is the default and that if running in Hyper-V mode, these instructions are not required - just capture inside the container.

workflow to capture and analyze traces for processes running inside these containers.

Important Limitation: Because ETW is a kernel-level feature and process-isolation containers

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kernel sessions specifically cannot be started from inside of a container. PerfView almost always captures a kernel session, so we should assume so here.

must be initiated from the host node.


Step 1: Capture a Trace on the Host Node (Required)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<h6>Step 1: Capture a Trace on the Host Node (Required)h6>
<h6>Step 1: Capture a Trace on the Host Nodeh6>

What happens if you don't use /EnableEventsInContainers: You will still capture all kernel
events (CPU sampling, context switches, etc.) for container processes, but you will miss user-mode events
like .NET garbage collection events, JIT events, exception events, and any custom EventSource events from
those processes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will still get user-mode events from processes running directly on the host node (outside of containers).

Then, inside the container, run the merge command to inject the necessary image identification data:

  • PerfViewCollect merge /ImageIDsOnly MyContainerTrace.etl.zip
  • Copy link
    Copy Markdown
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Note: PerfViewCollect needs to be built from source at https://github.com/microsoft/perfview. It is not currently shipped as a binary.

    What /ImageIDsOnly does: When you run merge with /ImageIDsOnly, PerfView reads through
    the trace and for each DLL that was loaded by processes in the trace, it looks up the DLL's unique
    identifier (signature/timestamp) and injects that information into the trace. This unique identifier

    Copy link
    Copy Markdown
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    It actually looks up the PDB signature.

    What happens if you don't run merge with /ImageIDsOnly: If you skip this step and later
    try to analyze the trace on another machine after the container is gone, PerfView will be unable to find
    the symbol files for DLLs that were loaded inside the container. Your stack traces will show addresses
    like "0x7ffe12345678" instead of method names for those DLLs. The managed code (.NET) symbols may still

    Copy link
    Copy Markdown
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    The stacks will just show the module name with a question mark. For example: MyAssembly!? instead of MyAssembly!MyClass.MyMethod.

    try to analyze the trace on another machine after the container is gone, PerfView will be unable to find
    the symbol files for DLLs that were loaded inside the container. Your stack traces will show addresses
    like "0x7ffe12345678" instead of method names for those DLLs. The managed code (.NET) symbols may still
    work since they come from NGEN PDBs or crossgen2 metadata, but native code (including the runtime itself)

    Copy link
    Copy Markdown
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Jitted code will still work, but nothing else will that comes from a binary inside of the container.

    will have missing symbols.

    Why run merge inside the container: The merge command needs access to the actual DLL

    Copy link
    Copy Markdown
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    The merger is a shared component and doesn't have access to look inside of containers, thus you must run it inside of the container(s).

    Co-authored-by: brianrob <6210322+brianrob@users.noreply.github.com>

    Copilot AI commented Dec 1, 2025

    Copy link
    Copy Markdown
    Contributor Author

    Copilot, please address these comments.

    Addressed all review comments in commit 15fe2b2.

    @brianrob

    Copy link
    Copy Markdown
    Member

    Copilot Post-Mortem:

    • As expected, it was easy to have copilot take my bullet points and write a more complete document.

    Mitchell (mitchellvette) pushed a commit to mitchellvette/zenvizor that referenced this pull request Jul 1, 2026
    Updated
    [Microsoft.Diagnostics.Tracing.TraceEvent](https://github.com/Microsoft/perfview)
    from 3.1.16 to 3.2.4.
    
    
    Release notes _Sourced from [Microsoft.Diagnostics.Tracing.TraceEvent's releases](https://github.com/Microsoft/perfview/releases)._ ## 3.2.4 ## Security This release contains security hardening fixes for a number of malformed-input parsing and path-traversal vulnerabilities: - Bounds-checking for malformed event payloads in the BPerf ULZ777 decompressor and event-record parser - Bounds-checking for malformed metadata in the GCDynamic, RegisteredTraceEventParser (TDH), Dynamic, and EventPipe V3 parsers - Bounds-checking for malformed PE CodeView and Resource directory entries - Path containment hardening for PDB extraction (zipped ETL + container PDBs), DiagSession resource extraction, R2R perf map writes, PdbScope module paths, and dynamic manifest writes - Path-traversal and command-execution hardening for Source Server lookups ## What's Changed * Update CsWin32 Package Version by @​brianrob in microsoft/perfview#2425 * Fix incorrect field offsets when parsing ETW events with fixed-count array fields by @​Copilot in microsoft/perfview#2427 * Retarget Native Profiler Builds To VS 2026 V145 Toolset by @​brianrob in microsoft/perfview#2428 * Stabilize XamlMessageBox UI-thread dispatch test by @​brianrob in microsoft/perfview#2430 **Full Changelog**: microsoft/perfview@v3.2.3...v3.2.4 ## 3.2.3 ## What's Changed * Upgrade Microsoft.Windows.CsWin32 to 0.3.209 (GHSA-ghhp-997w-qr28) by @​Copilot in microsoft/perfview#2409 * Enable Spectre mitigations and linker optimizations for EtwClrProfiler by @​danmoseley in microsoft/perfview#2410 * Fix 'unhanded' / 'occured' typos in UnhandledExceptionDialog body text by @​SAY-5 in microsoft/perfview#2413 * Fix GCStats failures on dotnet trace gc-verbose collections (#​2414) by @​cincuranet in microsoft/perfview#2415 * C entrypoint fixes by @​zachcmadsen in microsoft/perfview#2421 ## New Contributors * @​SAY-5 made their first contribution in microsoft/perfview#2413 **Full Changelog**: microsoft/perfview@v3.2.2...v3.2.3 ## 3.2.2 ## What's Changed * Fix PDB Symbol Resolution for Unmerged Windows Traces by @​brianrob in microsoft/perfview#2407 **Full Changelog**: microsoft/perfview@v3.2.1...v3.2.2 ## 3.2.1 ## Native and R2R Symbol Download and Parsing Now Available As of this release, if you capture a trace using [`dotnet-trace collect-linux`](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace#dotnet-trace-collect-linux) or [`record-trace`](https://github.com/microsoft/one-collect/tree/main/record-trace), **native and R2R symbols can now be downloaded and resolved at analysis time**. All .NET symbols (both native and R2R) are available on the Microsoft Symbol Server. Additionally, many Azure Linux symbol files are available on the Microsoft Symbol Server. For those targeting other distros, PerfView and TraceEvent are capable of pulling those symbol files from local directories by adding a local symbol path pointing to the files. Most of the work for this was completed in PerfView and TraceEvent 3.2.1 with the final required fixes present in this release. ## What's Changed * Optimize nettrace-to-TraceLog Conversion by @​brianrob in microsoft/perfview#2403 * Embed missing System.Text.Json transitive dependencies in PerfView by @​brianrob in microsoft/perfview#2404 **Full Changelog**: microsoft/perfview@v3.2.0...v3.2.1 ## 3.2.0 ## What's Changed * Fix Debug.Assert failures in SpeedScope tests and DynamicTraceEventParser by @​brianrob in microsoft/perfview#2368 * Add TraceParserGen.Tests project and fix code generation bugs by @​Copilot in microsoft/perfview#2308 * Update UsersGuide.htm by @​AftabAnsari10662 in microsoft/perfview#2370 * Strip .il and .ni suffixes from TraceModuleFile.Name by @​leculver in microsoft/perfview#2364 * Handle provider names that start with a numeric digit. by @​brianrob in microsoft/perfview#2369 * Dispose WebView2 controls before Environment.Exit to prevent finalizer crash by @​brianrob in microsoft/perfview#2371 * Refactor GetManifestForRegisteredProvider to use XmlWriter by @​Copilot in microsoft/perfview#2353 * docs: Add investigation guidance for JIT-inlined missing stack frames by @​Copilot in microsoft/perfview#2377 * Fix spurious BROKEN frame at top of Linux thread stacks in CPU Stacks viewer by @​Copilot in microsoft/perfview#2375 * Fix NRE in AddUniversalDynamicSymbol for invalid symbol address ranges by @​brianrob in microsoft/perfview#2376 * Add missing authority parameter to log by @​hoyosjs in microsoft/perfview#2379 * Replace individual code owners with microsoft/perfview-reviewers group by @​brianrob in microsoft/perfview#2381 * Fix Dynamic Symbol Resolution for Mappings Shared Across Multiple Processes in Universal Traces by @​brianrob in microsoft/perfview#2380 * Implement Symbol Demanglers for Linux Binaries by @​brianrob in microsoft/perfview#2383 * Fix NullReferenceException race condition in TraceLog.AllocLookup/FreeLookup by @​Copilot in microsoft/perfview#2387 * Add typed schema for AllocationSampled (EventID 303, .NET 10+) in ClrTraceEventParser by @​Copilot in microsoft/perfview#2388 * Add ElfSymbolModule for Parsing ELF Symbol Tables by @​brianrob in microsoft/perfview#2384 * Update BDN to latest version. by @​cincuranet in microsoft/perfview#2389 * Fixed overflow when working with large dumps by @​remilema in microsoft/perfview#2399 * Fix XamlMessageBox STA Threading Crash from Background Threads by @​brianrob in microsoft/perfview#2400 * Add ELF Symbol Resolution for Linux .nettrace Traces by @​brianrob in microsoft/perfview#2397 * Add Missing WCF Event Templates by @​brianrob in microsoft/perfview#2390 ## New Contributors * @​AftabAnsari10662 made their first contribution in microsoft/perfview#2370 * @​remilema made their first contribution in microsoft/perfview#2399 **Full Changelog**: microsoft/perfview@v3.1.30...v3.2.0 ## 3.1.30 ## What's Changed * doc: fix typos by @​chinwobble in microsoft/perfview#2359 * Fix SourceLink parsing to support both wildcard and exact path mappings by @​ivberg in microsoft/perfview#2355 * add horizontal scrolling to eventviewer by @​logangeorge01 in microsoft/perfview#2361 * Add SHA-384 and SHA-512 hash algorithm support for PDB checksums by @​Copilot in microsoft/perfview#2366 ## New Contributors * @​chinwobble made their first contribution in microsoft/perfview#2359 * @​logangeorge01 made their first contribution in microsoft/perfview#2361 **Full Changelog**: microsoft/perfview@v3.1.29...v3.1.30 ## 3.1.29 ## What's Changed * Warn users when circular buffer overflow causes missing type info in allocation views for selected processes by @​Copilot in microsoft/perfview#2326 * Special-Case BitMask Parsing by @​brianrob in microsoft/perfview#2327 * Refactor PEFile and PEHeader to use ReadOnlySpan exclusively with zero-copy buffer sharing by @​Copilot in microsoft/perfview#2317 * Fix cdbstack parser dropping last sample and missing metrics by @​Copilot in microsoft/perfview#2329 * Fix unhandled ArgumentOutOfRangeException when exporting FlameGraph with unrendered canvas by @​Copilot in microsoft/perfview#2339 * Add guidance for capturing ETW traces in Kubernetes pods by @​Copilot in microsoft/perfview#2344 * Fix merge command line order in kubernetes documentation by @​Copilot in microsoft/perfview#2346 * Fix GetRegisteredOrEnabledProviders() documentation claiming list is small by @​Copilot in microsoft/perfview#2348 * Fix duplicate stringTable elements in instrumentation manifest by @​Copilot in microsoft/perfview#2347 * Fix Histogram.AddMetric losing values after single-bucket to array transition by @​Copilot in microsoft/perfview#2337 * Fix clipboard copy formatting based on selection dimensions in Stack Viewer by @​Copilot in microsoft/perfview#2332 * Fix XML escaping in GetManifestForRegisteredProvider by @​Copilot in microsoft/perfview#2351 * Fix race condition in ProviderNameToGuid causing ERROR_INSUFFICIENT_BUFFER crashes by @​Copilot in microsoft/perfview#2357 **Full Changelog**: microsoft/perfview@v3.1.28...v3.1.29 ## 3.1.28 ## What's Changed * Add support for Boolean8 to NetTrace V6. by @​noahfalk in microsoft/perfview#2318 * Implement A Thread Time View for Universal Traces by @​brianrob in microsoft/perfview#2320 * Remove Incorrect Argument Description by @​brianrob in microsoft/perfview#2323 **Full Changelog**: microsoft/perfview@v3.1.26...v3.1.28 ## 3.1.26 Roll-up through 2025/10/10. * Only dispose non-null handles in `ETWTraceEventSource` [#​2291](microsoft/perfview#2291) * Small cleanup in `NettraceUniversalConverter` [#​2292](microsoft/perfview#2292) * Fix hyperlink focus visibility in dark mode and improve keyboard navigation [#​2295](microsoft/perfview#2295) * Gracefully handle invalid characters in `PATH` [#​2296](microsoft/perfview#2296) * Fix copying First/Last columns with pipe symbols to work in time range input [#​2304](microsoft/perfview#2304) ## 3.1.24 Roll-up through 2025/08/26. * Implement NuGet Central Package Version Management [#​2262] * Fix broken stacks warning for universal traces [#​2268] * Fix jitted code symbols in universal traces to show assembly names instead of memfd:doublemapper [#​2269] * Use themed background brush for menu and filter [#​2272] * Improve rendering and dark mode [#​2274] * Implement configurable symbol server authentication with /SymbolsAuth command line argument for PerfView and HeapDump [#​2278] * Add a themed dialog [#​2276] * Fix regression: "Goto Item in Callers/Callees" now accumulates across all threads [#​2284] * Fix parsing issues and add support for additional events to the Linux perf text file parser [#​2286] * Fix TraceLog live session RelatedActivityID/ContainerID corruption by preserving ExtendedData [#​2285] * NetTrace LabelList metadata overrides and metadata flushing [#​2281] * Fix NullReferenceException in ProviderBrowser.LevelSelected when deselecting level [#​2289] ## 3.1.23 Roll-up through 2025/07/11. - Fixed TraceEvent CaptureState API to support previously unsupported keyword configurations. [#​2222] - Added Exception Stacks view for .nettrace files to enhance exception diagnostics. [#​2223] - Corrected outdated documentation references to "GC Heap Alloc Stacks". [#​2224] - Fixed off-by-one error in P/Invoke buffer handling for Windows volume events. [#​2227] - Fixed broken links in the PerfView user guide. [#​2225] - Improved error handling by throwing when TdhEnumerateProviders fails, enabling better diagnostics. [#​2177] - Added AutomationProperties.Name to the Process Selection DataGrid for improved accessibility. [#​2239] - Fixed focus indicator visibility for hyperlinks in dark mode and high contrast themes. [#​2235] - Addressed NullReferenceException in Anti-Malware view. [#​2233] - Fixed WebView2 crash on close by implementing proper disposal pattern. [#​2230] - Added support for native AOT gcdumps, expanding compatibility with modern .NET workloads. [#​2242] - Fixed NVDA screen reader issue where Theme menu items did not announce selection state. [#​2237] - Extended PredefinedDynamicTraceEventParser to support dynamic events from additional sources. [#​2232] - Implemented MSFZ symbol format support in SymbolReader. [#​2244] - Removed usage of DefaultAzureCredential, simplifying authentication dependencies. [#​2255] - Added option to hide TimeStamp columns in the EventWindow View menu. [#​2247] - Fixed NVDA screen reader reporting incorrect list count for File menu separators. [#​2257] - Fixed unhandled exception when double-clicking in scroll bar area with no content. [#​2254] - Fixed universal symbol conversion for overlapping mappings. [#​2252] - Fixed TraceEvent.props to respect ProcessorArchitecture when RuntimeIdentifier is set. [#​2249] ## 3.1.22 Roll-up through 2025/06/04. - Added GC Heap Analyzer support for .nettrace files to enhance memory analysis workflows. [#​2216] - Introduced PredefinedDynamicTraceEventParser for known TraceLogging events, improving trace event parsing. [#​2220] - Enabled selection of process trees in the process selection dialog for multi-process analysis, allowing deeper inspection across related processes. [#​2195] - Implemented sorting for the Duration column in the process selection dialog using TotalDurationSeconds, improving usability. [#​2194] - Improved NetTrace parameter parsing for better command-line flexibility. [#​2200] - Fixed GetActiveSessionNames to handle ERROR_MORE_DATA, resolving session enumeration issues. [#​2196] - Fixed ObjectDisposedException when opening Net OS Heap Alloc Stacks, improving stability. [#​2212] - Fixed null reference exception in GenFragmentationPercent method, enhancing reliability. [#​2211] - Fixed TreeView auto-expansion when opening trace files. [#​2218] - Fixed StackViewer issue where "Set Time Range" reset "Goto Items by callees". [#​2208] - Fixed markdown table formatting when copying from the stack viewer. [#​2203] - Fixed TraceEvent NuGet package to exclude Windows-specific native DLLs. [#​2215] - Removed PDB generation for .NET Core assemblies using CrossGen, reducing build overhead. [#​2202] - Made symbol server timeout configurable and removed dead code in SymbolReader. [#​2209] - Changed help ribbons to use textblocks, enabling tab navigation. [#​2201] ## 3.1.21 Roll-up through 2025/05/02. - Change NetTrace format version support - Add /OSHeapMaxMB to set a max size for OS heap sessions - Implement Nettrace Support for Traces with Universal Providers - Implement R2R Symbol Lookup for Linux Traces - Fix NetTrace parsing for Start/Stop event names - Fix IndexOutOfRangeException in ProcessGlobalHistory ## 3.1.20 Roll-up through 2025/04/01. - Flamegraph and drill-in menu improvements - Performance improvements around unhandled event dispatch - Add configurable real-time delay in TraceLogEventSource - Don't queue another flush during a real-time ETW session if one is already in-process - Allow configuration of rundown providers for real-time EventPipe sessions - Fix stack handling for NetTrace V4 - Add multi-line view for events viewer - Misc accessibility fixes ## 3.1.19 Roll-up through 2025/01/30. - Added missing time information in the Raw XML View for GCStats. - Updated activity computation logic to support OpenTelemetry events. - Changed timestamp values to use QPC time based on UTC for Relogger. - Fixed issues with report command handling. - Addressed various POH-related issues. - Implemented file-size limitation for rundown using an ETW-based approach. ## 3.1.18 Roll-up through 2024/12/11. - Fixed `perfcollect` install script on Azure Linux 3. - Updated `System.Text.Json` to address dotnet/announcements#329. ## 3.1.17 Roll-up through 2024/11/08. - Numerous accessibility fixes to PerfView. This includes switching out the previous web browser plugin to use WebView2. - Breaking changes to FastSerialization to ensure that only expected types are deserialized. This addresses potential vulnerabilities during deserialization of untrusted input. Details at microsoft/perfview#2121. Commits viewable in [compare view](microsoft/perfview@v3.1.16...v3.2.4).
    Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Labels

    None yet

    Projects

    None yet

    Development

    Successfully merging this pull request may close these issues.

    Guidance on How to Capture Traces in Kubernetes Pods

    3 participants