Analyzing Windows heap usage with and without ETW

 

It has been a long time since I last wanted to discover if/where a program was “leaking” heap allocations. Most programs that I developed myself just performed some task and exited; heap allocations (from all sources, including Microsoft and other third party DLLs) probably rarely exceeded a few megabytes. I coded mostly with C# (garbage collected); most heap allocations directly under my control arose from native interop and I adopted an approach of releasing memory when it was “easy” and did not obscure the main intent of the code – otherwise I “intentionally” allowed the memory to leak.

I mention the above because I am a heavy user of Event Tracing for Windows (ETW) but I had hitherto no experience of using ETW (or, indeed, any other tool) to investigate heap usage. It was only when I tried to help with a problem/question in a technical forum that I had a need to understand heap usage. The question was whether the Windows Filtering Platform API FwpmNetEventEnum unavoidably leaks heap allocations.

The first approach that came to mind was to use the User-Mode Dump Heap (UMDH) utility from the Debugging Tools for Windows kit. However, the “current” version did not seem to work. Searching the web for explanations uncovered the following quotes for other users who had encountered the problem:

According to a Microsoft employee, this is a known problem. I quote: "Yeah. It's not working and I don't know when/if it will ever be."

I also quote an email I got from a Microsoft Support guy: "Anyway, I have confirmation it is broken. The dev team owning the exe knows about it and when they can get to fixing it they will."

Fortunately older versions of UMDH still work and it quickly became apparent that FwpmNetEventEnum does leak heap allocations. Most Fwpm* routines use RPC to the Base Filtering Engine (BFE) service to perform their function. Those Fwpm* APIs that return complex data structures mostly use a [allocate(all_nodes)] attribute in the MIDL ACF (Application Configuration File) so that the data can be freed with a single call to midl_user_free; however, that attribute was not applied to the RPC routine at the core of FwpmNetEventEnum. A subsequent call to FwpmFreeMemory just frees the top-level allocation and not the additional embedded allocations.

The absence of the [allocate(all_nodes)] attribute could be confirmed with tools that dump embedded RPC data structures; one example of a heap allocation back-trace that demonstrated that complex data structures were being allocated node-by-node was:

ntdll!RtlpAllocateHeapInternal+0x80B4E

fwpuclnt!MIDL_user_allocate+0x19

RPCRT4!NdrSafeAllocate+0x47

RPCRT4!Ndr64ComplexStructUnmarshall+0x72D

RPCRT4!Ndr64EmbeddedPointerUnmarshall+0x366

RPCRT4!Ndr64UnionUnmarshall+0x2D9

RPCRT4!Ndr64ComplexStructUnmarshall+0x5F4

RPCRT4!Ndr64pPointerLayoutUnmarshallCallback+0x234

RPCRT4!Ndr64ConformantArrayUnmarshall+0x21C

RPCRT4!Ndr64TopLevelPointerUnmarshall+0x40F

RPCRT4!Ndr64TopLevelPointerUnmarshall+0x59D

RPCRT4!Ndr64pClientUnMarshal+0x2A1

RPCRT4!NdrpClientCall3+0x40C

RPCRT4!NdrClientCall3+0xEB

fwpuclnt!FwpmNetEventEnum5+0x70

Heap Snapshots

I then turned my thoughts to understanding what type of bug could have been introduced into UMDH. There are several methods of obtaining the information needed to dump heap snapshot information (including heap allocation back-traces) about a process; the routines RtlQueryProcessDebugInformation and RtlQueryHeapInformation can both independently obtain the necessary information. UMDH seems to have taken a different approach and used the routine ReadProcessMemory and a knowledge of NTDLL internal data structures to gather the information.

The failing version of UMDH seems to have started using RtlQueryHeapInformation (with an HEAP_INFORMATION_CLASS value of HeapExtendedInformation (2)) to obtain information about heap allocations, but this information does not include any data that can be used to associate the allocation with a back-trace. There is, however, a HEAP_INFORMATION_CLASS value (5, let’s name it HeapStackTraceInformation) that returns information well suited for use by UMDH (i.e. includes information about allocated heap blocks and back-traces for the allocations).

The back-traces returned by RtlQueryHeapInformation for HeapStackTraceInformation come from a different source compared to the back-traces created and store when the Global Flag FLG_USER_STACK_TRACE_DB is set. The back-traces used by RtlQueryHeapInformation are enabled and disabled by RtlSetHeapInformation (also with a HEAP_INFORMATION_CLASS value of 5) or by creating a value named “FrontEndHeapDebugOptions” under the Image File Execution Options (IFEO) key for an image; this value can be set by the Windows Performance Recorder (WPR) command “wpr -snapshotconfig heap –name […]” (“wpr -snapshotconfig heap –pid […]” effectively calls RtlSetHeapInformation).

When comparing the two versions of the back-trace information for a given allocation, they mostly just differ in the first frame:

HeapStackTraceInformation:

ntdll!RtlpAllocateHeapInternal+0x80b49:

e8528d0500      call    ntdll!RtlpHpStackTraceAddStack

 

FLG_USER_STACK_TRACE_DB:

ntdll!RtlpAllocateHeapInternal+0x809dd:

e8ac1cffff      call    ntdll!RtlpCallInterceptRoutine

The back-traces can also differ in the depth of the back-trace captured and stored (HeapStackTraceInformation can save more frames).

“wpr -singlesnapshot heap […]” uses EnableTraceEx2 to send an EVENT_CONTROL_CODE_CAPTURE_STATE to the Microsoft-Windows-Heap-Snapshot provider, using the EnableFilterDesc field of the EnableParameters parameter to select the “pids”. This causes RtlQueryHeapInformation with HeapStackTraceInformation to be executed in the target processes with the output being broken into chunks and logged into the trace session. Windows Performance Analyzer (WPA) can reassemble, analyze and display this data in a “Heap Snapshot” graph.

Heap Events

WPR provides another heap related command: “wpr -heaptracingconfig […]”. This command creates/sets another value under IFEO – namely TracingFlags. These flags enable aspects of the User Mode Global Logger (UMGL), including events generated by the WMI HeapTraceProvider; this provider generates events for individual heap events (HeapRangeCreate, HeapRangeReserve, HeapRangeRelease, HeapRangeDestroy, HeapCreate, HeapAllocation, HeapReallocation, HeapDestroy, HeapFree and more) and StackWalk back-traces can be configured for selected event types. WPA knows how to analyze and display these events too (in various graphs in the Memory category).

The instrumentation for these events is obviously embedded in many NTDLL heap routines; for the HeapAllocation event, the instrumentation is embedded close to the heap stack tracing calls:

ntdll!RtlpAllocateHeapInternal+0x80aec:

e817a30500      call    ntdll!RtlpLogHeapAllocateEvent

If a process was started without heap tracing enabled via IFEO, heap tracing can still be enabled by directly setting the heap tracing bit in the _PEB.TracingFlags field (perhaps via a debugger); there does not seem to be any API that performs this function.

Event Tracing for Windows (ETW) Data in Bug Check Secondary Dump Data

Secondary Dump Data seems to be a seldom discussed topic; a useful introduction is the “Inside Show” episode “Bugcheck Secondary Dump Data” by Andrew Richards and Chad Beeder (the main presenters of the “Defrag Tools” shows).

The Secondary Dump Data mechanism can be used by all developers of kernel-mode software (including Microsoft). The information included in the dump is whatever makes sense to the developer, so it can be difficult for someone else to interpret. Microsoft has added some debugger extensions to dump some of their secondary dump data. The table below shows some of the GUIDs used to identify items of secondary dump data and the debugger extension commands that displays them (where known).

GUID Symbol	Debugger Extension Command
mssmbios!SMBiosGuidAcpi	!sysinfo gbl
mssmbios!SMBiosGuidBios	!sysinfo cpuinfo cpuspeed cpumicrocode
mssmbios!SMBiosGuidRegisters	!sysinfo registers
mssmbios!SMBiosGuidSMBios	!sysinfo machineid
nt!EtwSecondaryDumpDataGuid	!wmitrace.strdump !wmitrace.logsave
nt!GUID_TRIAGEDUMP_DATA
nt!PopBlackBoxAcpiGuid	!ext.blackboxacpi
nt!PopBlackBoxBsdGuid	!ext.blackboxbsd
nt!PopBlackBoxCodeIntegrityGuid
nt!PopBlackBoxCrashedProcessGuid	!ext.blackboxcrashedprocess
nt!PopBlackBoxDxgDisplayGuid	!ext.blackboxdxg
nt!PopBlackBoxExplorerCoreStartupGuid
nt!PopBlackBoxExplorerLogonTasksGuid
nt!PopBlackBoxNtfsGuid	!ext.blackboxntfs
nt!PopBlackBoxPdcLockGuid	!ext.blackboxpdclock
nt!PopBlackBoxPnpDelayedRemoveWorkerGuid	!ext.blackboxpnpdelayedremoveworker
nt!PopBlackBoxPnpDeviceCompletionQueueGuid	!ext.blackboxpnpdevicecompletionqueue
nt!PopBlackBoxPnpEventWorkerGuid	!ext.blackboxpnpeventworker
nt!PopBlackBoxPnpGuid	!ext.blackboxpnp
nt!PopBlackBoxPoIrpGuid
nt!PopBlackBoxPoPepWorkOrderGuid	!ext.blackboxpopepworkorder
nt!PopBlackBoxPoPowerWatchdogGuid	!ext.blackboxpowerwatchdog
nt!PopBlackBoxScmGuid
nt!	!ext.blackboxstoremanager
nt!PopBlackBoxUsoCommitGuid
nt!PopBlackBoxWheaGuid
nt!PopBlackBoxWinLogonGuid
nt!PopBlackBoxWinLogonNotifyGuid
pci!PCI_CFG_RECORD_GUID	!pci
Wdf01000!WdfDumpGuid	!wdfkd.wdfcrashdump

 

The debugger (extension) commands “!rcdrkd.rcdrenumtag” and “.enumtag” show the tags and data included in the secondary dump data “stream” of a dump file.

Of particular interest to me is the EtwSecondaryDumpData; ETW trace sessions started with the flag EVENT_TRACE_ADDTO_TRIAGE_DUMP set in the logging mode will have their “in-memory” trace buffers added to the secondary dump data. On my Windows 11 system, the Microsoft sessions named “EventLog-System”, “Microsoft-Windows-Rdp-Graphics-RdpIdd-Trace” and “WiFiSession” have this flag set, as do some third party sessions (e.g. “IntelRST”).

In principle, trace files for the sessions can be extracted using debugger extension commands; in practice, a few bugs currently make this difficult.

EtwSecondaryDumpData starts with some values that can be used to reconstruct the TRACE_LOGFILE_HEADER of every trace in the system (values for ProviderVersion, TimerResolution, CpuSpeedInMHz,  BootTime and PerfFreq). Following this, for each session that has the EVENT_TRACE_ADDTO_TRIAGE_DUMP flag set, a summary of the WMI_LOGGER_CONTEXT (dt nt!_WMI_LOGGER_CONTEXT) for the session is added to the data, including the “Logger Name”.

In the WMI_LOGGER_CONTEXT structure, LoggerName is a UNICODE_STRING structure. The documentation for the “Length” member of UNICODE_STRING says “Specifies the length, in bytes, of the string pointed to by the Buffer member, not including the terminating NULL character, if any” (my emphasis). Older versions of Windows seem to interpret “Length” as number of Unicode characters (UTF-16) and copy twice the length of the Logger Name to the secondary dump data. The debugger extension “!wmitrace.strdump” follows this example and expects the same (double) length; this works on old secondary dump data but fails silently on secondary dump data of newer Windows versions which copy the correct name length (displays nothing beyond the first session name).

Note that “!wmitrace.strdump” detects whether a dump is a triage dump or a larger dump; if the dump is a triage dump (DUMP_TYPE_TRIAGE) then “!wmitrace.strdump” uses the secondary dump data but, if the dump is larger, then the full kernel WMI_LOGGER_CONTEXT structures are used.

The second bug (at the time of writing) is in the wmitrace routine that reads trace buffers (dt nt!_WMI_BUFFER_HEADER); the routine seems to expect that full buffers will be read (length = WMI_BUFFER_HEADER.BufferSize) although its parameters allow for sections of a buffer to be read. Since the “!wmitrace.logsave” debugger extension tries to read buffer sections, it normally fails.

It was possible to verify that the debugger extensions could work (by debugging the debugger and fixing-up the issues) but, in practice, it is currently more comfortable (and insightful) to re-implement the functionality.

This feature set (marking selected trace sessions with EVENT_TRACE_ADDTO_TRIAGE_DUMP and adding such session trace buffers to triage dumps) is very useful in remote support/diagnostic scenarios where exchanges of full dumps are undesirable (because of confidentiality and size issues) but some contextual/historical information is essential to understand the state of the limited amount of kernel memory in a triage dump.

Note that triage dumps that are not created from a bug check (e.g. triage dumps created by commands such as “kdbgctrl -td <pid> <file>”) do not include secondary dump data.

Event Tracing for Windows (ETW) Data in Bug Check Primary Minidump Data

There is a very limited amount of ETW data in the primary data of a small memory dump (minidump). The debugger extension commands “!wmitrace.dumpmini” and “!wmitrace.dumpminievent” are intended to dump this data; similar to some of the commands previously mentioned, these commands often choose/use one of the 3 “offset” fields in WMI_BUFFER_HEADER which indicates that the buffer is empty. As a workaround, the commands “!wmitrace.buffer poi(nt!KiCurrentEtwBufferBase)” and “!wmitrace.buffer poi(nt!KiCurrentErrLogBufferBase)” give a flavor of the information available.

“!wmitrace.dumpmini” displays the current buffer of the “Circular Kernel Context Logger” session for the current processor – the “used” portion of this buffer is explicitly added to the kernel memory dumped by a small memory dump. “!wmitrace.dumpminievent” displays the current buffer of the “EventLog-System” session if a “current” buffer was identified at the time of the dump (normally there is/was no “current” buffer for this session) and added to the kernel memory dumped.

UPDATE

While thinking about troubleshooting strategies for a problem, I noticed two more secondary dump data items to be interesting for me: nt!GUID_TRIAGEDUMP_DATA and ipt!GUID_IPT.

Triage Dump Data

The “triage dump data” secondary dump data (with GUID nt!GUID_TRIAGEDUMP_DATA) is a relatively simple list of kernel address ranges (base and size) grouped by “component” (a free-form Latin 1 string). This data can be attached to most dump types. In a “large” dump, it indicates which ranges should be added to a “carved” minidump (a minidump created from a larger dump).

The topic that I was thinking about was how to access the buffers that were being used by the Intel Processor Trace (IPT) facility at the time of the crash/dump; the IPT driver adds ranges describing these buffers to the triage dump data (using the component name “IPT”) making them easy to identify. This behaviour also makes it possible to share just carved minidumps when remotely diagnosing a problem and wanting access to the last recorded IPT entries.

Intel Processor Trace (IPT) Data in Bug Check Secondary Dump Data

If IPT is active at the time of a dump (including a “live dump”) then IPT secondary dump data is added to the dump (using the GUID ipt!GUID_IPT). The header of the secondary dump data is an IPT_TRACE structure and two TraceType values appear to be used: one to describe “thread” traces and one to describe “core” traces.

The thread trace secondary data is mostly just a simple summary of the number of threads being traced.

The core trace secondary data is more useful: it contains the kernel address ranges of the trace buffers plus the values of the relevant Intel Model-Specific Registers (MSRs) such as IA32_RTIT_OUTPUT_MASK_PTRS and IA32_RTIT_STATUS.

The usefulness of the trace data is still an open question for me. I have only experimented with the traces in conjunction with a “live” dump and the process of creating the dump is traced, meaning that the trace of the moments before the “dump” will almost certainly be overwritten. It is conceivable that, in the event of crash, the IPT mechanism will be stopped quickly (thus preserving the moments leading up to the crash).

Windows Filtering Platform and Window Service Hardening Rules

The Windows Filtering Platform (WFP) is an important Windows system component that I had only ever endeavoured to understand in sufficient depth to meet current needs.

The Microsoft documentation says: “Windows Filtering Platform (WFP) performs its tasks by integrating the following basic entities: Layers, Filters, Shims, and Callouts.”

Use (and management) of rules in Windows Defender Firewall required an understanding of WFP filters; interpreting the data captured by Microsoft Message Analyser benefitted from understanding WFP layers and callouts; experimenting with IPsec, VPN and DirectAccess benefitted from understanding WFP layers, callouts and provider contexts.

One aspect of WFP that I dismissed/ignored as just a grouping mechanism was WFP sublayers; their role in classification and filter arbitration is described in the Microsoft documentation, but I never previously read this closely enough.

I wrote about Network Discovery last year and was surprised and embarrassed when I noticed that the list of discovered computers under Windows 11 was incomplete for reasons that I could not explain – the local computer (which had previously always been in the list) was not present. Initially, I just quickly dismissed this as a “by design” decision until I noticed a correlation between the process of network discovery and WFP packet drop events. Here is the output of the “netsh wfp show netevents” command for the drop event:

<header>
       <timeStamp>2022-04-09T09:14:32.144Z</timeStamp>
       <flags numItems="9">
              <item>FWPM_NET_EVENT_FLAG_IP_PROTOCOL_SET</item>
              <item>FWPM_NET_EVENT_FLAG_LOCAL_ADDR_SET</item>
              <item>FWPM_NET_EVENT_FLAG_REMOTE_ADDR_SET</item>
              <item>FWPM_NET_EVENT_FLAG_LOCAL_PORT_SET</item>
              <item>FWPM_NET_EVENT_FLAG_REMOTE_PORT_SET</item>
              <item>FWPM_NET_EVENT_FLAG_APP_ID_SET</item>
              <item>FWPM_NET_EVENT_FLAG_USER_ID_SET</item>
              <item>FWPM_NET_EVENT_FLAG_IP_VERSION_SET</item>
              <item>FWPM_NET_EVENT_FLAG_PACKAGE_ID_SET</item>
       </flags>
       <ipVersion>FWP_IP_VERSION_V6</ipVersion>
       <ipProtocol>17</ipProtocol>
       <localAddrV6.byteArray16>::1</localAddrV6.byteArray16>
       <remoteAddrV6.byteArray16>::1</remoteAddrV6.byteArray16>
       <localPort>50602</localPort>
       <remotePort>3702</remotePort>
       <scopeId>0</scopeId>
       <appId>
       <data>5c006400650076006900630065005c0068006100720064006400690073006b0076006f006c0075006d00650033005c00770069006e0064006f00770073005c00730079007300740065006d00330032005c0073007600630068006f00730074002e006500780065000000</data>
       <asString>\.d.e.v.i.c.e.\.h.a.r.d.d.i.s.k.v.o.l.u.m.e.3.\.w.i.n.d.o.w.s.\.s.y.s.t.e.m.3.2.\.s.v.c.h.o.s.t...e.x.e...</asString>
       </appId>
       <userId>S-1-5-19</userId>
       <addressFamily>FWP_AF_INET6</addressFamily>
       <packageSid>S-1-0-0</packageSid>
       <enterpriseId/>
       <policyFlags>0</policyFlags>
       <effectiveName/>
</header>
<type>FWPM_NET_EVENT_TYPE_PUBLIC_CLASSIFY_DROP</type>
<classifyDrop>
       <filterId>69067</filterId>
       <layerId>46</layerId>
       <reauthReason>0</reauthReason>
       <originalProfile>0</originalProfile>
       <currentProfile>0</currentProfile>
       <msFwpDirection>MS_FWP_DIRECTION_OUT</msFwpDirection>
       <isLoopback>true</isLoopback>
       <vSwitchId/>
       <vSwitchSourcePort>0</vSwitchSourcePort>
       <vSwitchDestinationPort>0</vSwitchDestinationPort>
</classifyDrop>
<internalFields>
       <internalFlags numItems="1">
              <item>FWPM_NET_EVENT_INTERNAL_FLAG_FILTER_ORIGIN_SET</item>
       </internalFlags>
       <capabilities/>
       <fqbnVersion>0</fqbnVersion>
       <fqbnName/>
       <terminatingFiltersInfo numItems="4">
              <item>
                     <filterId>67000</filterId>
                     <subLayer>FWPP_SUBLAYER_INTERNAL_FIREWALL_APP_ISOLATION</subLayer>
                     <actionType>FWP_ACTION_PERMIT</actionType>
              </item>
              <item>
                     <filterId>66827</filterId>
                     <subLayer>FWPP_SUBLAYER_INTERNAL_FIREWALL_QUARANTINE</subLayer>
                     <actionType>FWP_ACTION_PERMIT</actionType>
              </item>
              <item>
                     <filterId>69067</filterId>
                     <subLayer>FWPP_SUBLAYER_INTERNAL_FIREWALL_WSH</subLayer>
                     <actionType>FWP_ACTION_BLOCK</actionType>
              </item>
              <item>
                     <filterId>66219</filterId>
                     <subLayer>FWPP_SUBLAYER_INTERNAL_FIREWALL_WF</subLayer>
                     <actionType>FWP_ACTION_PERMIT</actionType>
              </item>
       </terminatingFiltersInfo>
       <filterOrigin>WSH Default</filterOrigin>
       <interfaceLuid>6755399457832960</interfaceLuid>
</internalFields>

The section highlighted in yellow was new to me and particularly intriguing was the “terminatingFiltersInfo” data. It only became clear later that what this showed was all of the sublayers in layer 46 (FWPS_LAYER_ALE_AUTH_RECV_ACCEPT_V6) that contained one or more filters that matched the packet. Within each sublayer the filters are evaluated and the filter that delivers the “terminating” action (based on filter weight, action, rights, etc. as discussed in the Filter Arbitration documentation) for that sublayer is reported.

Comparing the filters of Windows 10 and 11 systems gave part of the answer why the local computer was missing from the list of computers. The equivalent for the terminating filter for the FWPP_SUBLAYER_INTERNAL_FIREWALL_APP_ISOLATION sublayer in Windows 11 (named “AppContainerLoopback”) is defined in the FWPM_SUBLAYER_MPSSVC_WSH sublayer under Windows 10. The (blocking) terminating filter in the FWPP_SUBLAYER_INTERNAL_FIREWALL_WSH sublayer is named “WSH Default Inbound Block”.

This means that in Windows 10, the “AppContainerLoopback” filter can (and does, via weighting) override the “WSH Default Inbound Block” filter because they are in the same sublayer; this is not true for Windows 11.

This change of sublayer for the “AppContainerLoopback” loopback filter explains the difference in behaviour between Windows 10 and 11 but still leaves a mystery: there are 7 WSH rules/filters for the “Function Discovery Provider Host” (fdphost) service and one, named “Allow inbound UDP traffic to fdphost port 3702”, is possibly intended to allow responses to the WS-Discovery multicast probes to be received.

Some more detailed tracing is needed to form a hypothesis for this behaviour. The trace command that I used was:

pktmon start --trace --provider Microsoft-Windows-WFP --provider Microsoft-Windows-TCPIP --keywords 0x300408080 --level 17 --provider "TCPIP Service Trace" --keywords 0x17100 --level 6 --file-name why.etl

Provider Microsoft-Windows-WFP is an obvious choice and the Microsoft-Windows-TCPIP provider with the keywords ut:TcpipDiagnosis, ut:AleRemoteEndpoint, ut:Loopback, ut:SendPath, ut:ReceivePath limits the (verbose) output of that provider to the more relevant events. These two providers give background information about what is happening at any particular time and the "TCPIP Service Trace" with keywords WFP_TRACE_BASE, WFP_TRACE_FE, WFP_TRACE_STM, WFP_TRACE_ALE, NETIO_TRACE_TUNNEL provides the details.

"TCPIP Service Trace" is a WPP (Windows Software Trace Preprocessor) provider, so its data is difficult to interpret without the corresponding private .pdb file. Here is an indication of the type of information in the trace data:

This shows the packet being sent, no existing flow found and each of the filters in the layer being tested against the packet:

The data made available to the filters is also logged, as is the final result:

Once the packet has passed the outbound filters, one can see that the packet is looped back and inbound filtering is started:

Finally, one can see that the packet is dropped:

The filter responsible for the drop (“WSH Default Inbound Block”) only has one filter condition and that is a match against FWPM_CONDITION_ALE_USER_ID. The information used for this condition is a self-relative TOKEN_ACCESS_INFORMATION structure and this is included in the trace (as a binary blob).

Formatting that blob shows something like this:

SidHash Offset 0x58
SidHash 0x00000000 S-1-5-19
SidHash 0x00000060 S-1-16-16384
SidHash 0x00000007 S-1-1-0
SidHash 0x00000007 S-1-5-32-545
SidHash 0x00000007 S-1-5-6
SidHash 0x00000007 S-1-2-1
SidHash 0x00000007 S-1-5-11
SidHash 0x00000007 S-1-5-15
SidHash 0x0000000E S-1-5-80-364023826-931424190-487969545-1024119571-74567675
SidHash 0xC000000F S-1-5-5-0-11718649
SidHash 0x00000007 S-1-2-0
SidHash 0x00000007 S-1-5-32-3167453650-624722384-889205278-321484983-714554697-3592933102-807660695-1632717421
SidHash 0x00000007 S-1-5-32-383293015-3350740429-1839969850-1819881064-1569454686-4198502490-78857879-1413643331
SidHash 0x00000007 S-1-5-32-2035927579-283314533-3422103930-3587774809-765962649-3034203285-3544878962-607181067
SidHash 0x00000007 S-1-5-32-3659434007-2290108278-1125199667-3679670526-1293081662-2164323352-1777701501-2595986263
SidHash 0x00000007 S-1-5-32-11742800-2107441976-3443185924-4134956905-3840447964-3749968454-3843513199-670971053
SidHash 0x00000007 S-1-5-32-3523901360-1745872541-794127107-675934034-1867954868-1951917511-1111796624-2052600462
SidHash 0x00000007 S-1-5-32-1488445330-856673777-1515413738-1380768593-2977925950-2228326386-886087428-2802422674
RestrictedSidHash Offset 0x4A0
Privileges Offset 0x800
Privilege 0x00000003 0x0000000000000017
Privilege 0x00000003 0x000000000000001D
AuthenticationId 0x3E5
TokenType TokenPrimary
ImpersonationLevel SecurityAnonymous
MandatoryPolicy NO_WRITE_UP, NEW_PROCESS_MIN
Flags 0x1002800
AppContainerNumber 0
CapabilitiesHash Offset 0x5B0
SecurityAttributes Offset 0x6C0
SecurityAttributes Name="TSA://ProcUnique" Type=2 Flags=0x0 Count=2 X1=0x41
  Value[0]=245
  Value[1]=11718747 

The S-1-5-80 (service SID) is the service SID of the receiving process (“fdphost” (Function Discovery Provider Host)).

fdphost sends a WS-Discovery probe to a multicast address and UDP port 3702, but it sends the probe from its randomly assigned local port number; replies to the probes are sent back to this port number. Non-loopback replies are permitted by an ALE multicast flow but, for currently unknown reasons, loopback replies are subjected to a full “classification” – which results in a drop decision. My hypothesis is that no attempt is made to match loopback packets against existing ALE multicast flows.

I searched the web for mentions of the local system not appearing in the list of discovered computers and found none. I also asked on the Microsoft Q&A website and received one report that the issue did not appear to be present – so the hypothesis is unconfirmed (or even “in doubt”) at the moment…

PktMon

Judging by Web search results, Windows 10 has included a new network traffic capturing mechanism since October 2018; however, two and a half years later, it still seems to be largely unknown (I only discovered it a few days ago).

Microsoft provides and supports a “classic” NDIS Filter traffic capturing mechanism (NdisCap, its associated Microsoft-Windows-NDIS-PacketCapture ETW provider and a PowerShell cmdlet Add-NetEventPacketCaptureProvider) and previously also supported a Windows Filtering Platform capture mechanism (WFPCapture, its associated Microsoft-Pef-WFP-MessageProvider ETW provider and a PowerShell cmdlet Add-NetEventWFPCaptureProvider). The new mechanism (PktMon and its associated Microsoft-Windows-PktMon ETW provider) does not yet have a PowerShell cmdlet to add it to ETW tracing sessions.

According to the PktMon home page, “[PktMon] is especially helpful in virtualization scenarios, like container networking and SDN, because it provides visibility within the networking stack”. The mechanism that allows PktMon to intercept a packet at various points in its transition through the network stack are additional “hooks” introduced into NDIS.sys. Some typical stack traces of the points at which PktMon is invoked are:

PktMon!PktMonPacketLogCallback+0x19
ndis!PktMonClientNblLog+0xbd
ndis!PktMonClientNblLogNdis+0x2b
ndis!ndisCallSendHandler+0x3ca4b
ndis!ndisInvokeNextSendHandler+0x10e
ndis!NdisSendNetBufferLists+0x17d

PktMon!PktMonPacketLogCallback+0x19
ndis!PktMonClientNblLog+0xbd
ndis!PktMonClientNblLogNdis+0x2b
ndis!ndisMIndicateNetBufferListsToOpen+0x3e95c
ndis!ndisMTopReceiveNetBufferLists+0x1bd
ndis!ndisCallReceiveHandler+0x61
ndis!ndisInvokeNextReceiveHandler+0x1df
ndis!ndisFilterIndicateReceiveNetBufferLists+0x3be91
ndis!NdisFIndicateReceiveNetBufferLists+0x6e

PktMon can be seen as an improvement on NdisCap. The main advantage (in my opinion) is that PktMon can be loaded and started without requiring rebinding of the network stack. As I have mentioned in other articles, rebinding the network stack can, under unfortunate circumstances, be a risky undertaking. The new ability to intercept packets at various points in the network stack is something that I have never personally had a need to use but is probably welcomed by those who have had difficulty in diagnosing network problems in “virtualization scenarios”.

One thing that PktMon cannot do is to trace loopback traffic, since the Windows loopback implementation does not use NDIS (WFP mechanisms and raw sockets can capture such traffic).

There are 3 main components of PktMon: the driver (PktMon.sys), a DLL (PktMonApi.dll) and an executable (PktMon.exe).

PktMon.sys

PktMon.sys is the core component. It is controlled via a small set of IOCTLs (to start, stop and query a capture; add, remove and list packet filters; list traceable components; reset trace counters) and the keywords used in the ETWENABLECALLBACK (Config, Rundown, NblParsed, NblInfo and Payload).

The information in the list of traceable components will seem familiar to anyone who has used the kernel debugger extensions “!ndiskd.miniports”, “!ndiskd.protocols” and “!ndiskd.filters”. The list of components is not only available via the IOCTL but is also included (in a different form) at the end of an ETW trace if the “Rundown” keyword is enabled.

The packet filtering possibilities are of the address/protocol-type/port type rather than what Microsoft sometimes calls OLP (Offset value, bit Length, and value Pattern). The output of the command “pktmon filter add help” accurately reflects the filtering possibilities. The filtering mechanism does not allow “negative” conditions to be expressed – for example, one can’t specify “ignore RDP” (as one might wish to do if one is logged onto a system via RDP).

Similar to both NdisCap and WFPCapture, PktMon must be explicitly loaded and/or started before it can generate any events; just starting an ETW trace session containing Microsoft-Windows-PktMon is not enough to capture trace data.

PktMonApi.dll

PktMonApi.dll currently has 9 exports, which are mostly just simple wrappings around IOCTLs to PktMon.sys:

PktmonAddFilter
PktmonGetComponentList
PktmonGetFilterList
PktmonGetStatus
PktmonRemoveAllFilters
PktmonResetCounters
PktmonStart
PktmonStop
PktmonUnload

PktMonApi.dll is not used by PktMon.exe (which contains its own simple wrappings around the IOCTLs).

PktMon.exe

PktMon.exe has several facets: it can configure and control PktMon.sys via its IOCTLs, it can manage ETW trace sessions, it can extract information from the Microsoft-Windows-PktMon ETL and save it in various formats (including “pcapng”), and it can perform “tcpdump” style simple formatting of packets captured and display in real-time.

As mentioned, PktMon.sys must be explicitly managed in order for Microsoft-Windows-PktMon to capture data. It would be ideal (for me) if Microsoft-Windows-PktMon could just be included in a Windows Performance Recorder (WPR) Profile along with other providers and use of advanced ETW features (such as stack traces, SID information, etc.). Since that is not possible, one must be content with the limited ETW configuration options of PktMon.exe (provider, keywords and level) – a similar situation to that with NdisCap and “netsh trace”.

The PktMon home page currently says “Packet drops from Windows Firewall are not visible through Packet Monitor yet” (my emphasis of “yet”). I would not expect Windows Firewall drop detection to be included in PktMon.sys, since they deal with different technologies. By including both Microsoft-Windows-PktMon and Microsoft-Windows-WFP in ETW trace sessions, one can see both the drops types detected by PktMon and the drops caused by Windows Firewall. Perhaps the “yet” is just a nod to future improvements in the PktMon.exe user interface to present a unified view of the output of the two providers.

The configuration defaults of PktMon.exe cause packets to be logged at all interception points (--comp all); if the intent of a capture is just to analyse the traffic in a tool like Wireshark, this can cause each packet to be repeated twenty or more times in the network trace. One can tackle this by selecting specific components when exporting captured data to the pcapng format, but I prefer to use “PktMon start” with the “--comp nics” qualifier.

Microsoft Message Analyzer

Microsoft Message Analyzer (MMA) was discontinued more-or-less contemporaneously with the introduction of PktMon and the PktMon team provides no support for MMA. The OPN (Open Protocol Notation) below allows the ETL output of PktMon to be viewed comfortably in MMA (if one still has a copy installed).

The OPN is short and it works (for me), but it includes some “design” decisions and is incomplete (no support for the MBB (Mobile BroadBand) NDIS medium type, for example, and probably in many unknown/unanticipated ways).

module PktMon;

using Microsoft_Windows_PktMon;
using Standard;
using Ethernet;
using WiFi;

autostart actor PktMonPayload(ep_Microsoft_Windows_PktMon e)
{
    process e accepts m:Event_160 where m.PacketType == 1
    {
                dispatch endpoint Ethernet.Node accepts BinaryDecoder<Ethernet.Frame[m.LoggedPayloadSize < m.OriginalPayloadSize]>(m.Payload) as Ethernet.Frame;
    }

    process e accepts m:Event_160 where m.PacketType == 2
    {
                DecodeWiFiMessageAndDispatch(m.Payload);
    }
}

Saving this OPN in a file named %LOCALAPPDATA%\Microsoft\MessageAnalyzer\OPNAndConfiguration\OPNForEtw\CoreNetworking\PktMon.opn (for example) and restarting MMA should enable the functionality (MMA may take some time to completely start while it recompiles various OPN files).

Microsoft_Windows_PktMon

The Microsoft_Windows_PktMon ETW provider defines 5 keywords:

1.       Config: PktMon.exe help says “Internal Packet Monitor errors”; I have never observed any.

2.       Rundown: this causes the list of components to be logged to the ETL when PktMon is stopped.

3.       NblParsed: this causes address, protocol type, port, etc. information for each packet to be logged. The same information could be extracted from the binary payload (if present), but it is not trivial to do this because of various options at each layer (data link, network, transport).

4.       NblInfo: this seems to be a superset of the information logged as TcpipNlbOob by the Microsoft-Windows-TCPIP provider at level 17 (uninteresting for most people).

5.       Payload: this causes the raw data of the packet to be logged. The data can be truncated, if desired (truncation length information is included in the start IOCTL to PktMon.sys).

pktmon etl2pcap

Converting pktmon packet events to PCAPNG format is, in principle, relatively straightforward. There is a GitHub Microsoft repository with the code of a utility that performs the slightly more difficult task of converting “netsh trace” packet data to PCAPNG format (https://github.com/microsoft/etl2pcapng).

Perhaps surprisingly, the “pktmon etl2pcap” command (“Convert pktmon log file to pcapng format”) only produces a useful PCAPNG file if the packets were captured on an Ethernet/802.3 link. If packets were captured on a WiFi/802.11 link, the resulting PCAPNG packet is recorded as having been captured on an Ethernet link; since the datalink headers are different in content and length, a tool like Wireshark “decodes” the packet data incorrectly. If most of the packets in the capture were obtained from a WiFi/802.11 link, the command “editcap -T ieee-802-11 <infile> <outfile>” should make the PCAPNG file usable.

Apart from correctly identifying 802.11 frames as such (LINKTYPE_IEEE802_11), there is one other additional step that might be needed when saving 802.11 packets to PCAPNG format: the “protected” flag in the 802.11 frame control field might need to be cleared. The “protected” flag indicates whether the frame was encrypted; packets carrying network layer data are protected/encrypted but, by the time received packets have reached the packet capture hooks, the protected/encrypted content has been decrypted – however the captured 802.11 packet header (frame control, etc.) is not necessarily updated (probably depends on the network interface driver). If the protected bit is not cleared when saving the capture then, when the capture is loaded into Wireshark, the packet is assumed to still be protected and is displayed as such (no attempt is made to decode the “encrypted” portion of the packet).

pktmon start --trace

PktMon can control (start/stop) other ETW trace providers, as can “netsh trace”, logman and wpr (Windows Performance Recorder) amongst others. However, PktMon differs from the other controllers in the default values for “keywords” and “level” passed to the providers. By default, most providers use “maximal” values for keywords and level but PktMon uses a value of 0xFFFFFFFF for the keywords value (which is actually a 64-bit value, so the high 32 bits are set to zero) and 4 for the level (the maximum value is 255 and Microsoft-Windows-TCPIP, for example, logs some events at level 17).

Although the help/documentation (“pktmon start help”) mentions this, I have been caught out more than once puzzling over why certain expected events were missing from a trace.

New Packet Monitor API (Windows 11 24H2, Windows Server 2025)

PktMonApi.dll now exports additional, documented routines:

PacketMonitorAddCaptureConstraint
PacketMonitorAddSingleDataSourceToSession
PacketMonitorAttachOutputToSession
PacketMonitorCloseRealtimeStream
PacketMonitorCloseSessionHandle
PacketMonitorCreateLiveSession
PacketMonitorCreateRealtimeStream
PacketMonitorEnumDataSources
PacketMonitorInitialize
PacketMonitorSetSessionActive
PacketMonitorUninitialize

The documentation highlights “Multisession” and “Packet-streaming” as new capabilities. The new capabilities are enabled via “Controlled Feature Rollout” (CFR) for the feature named "UxConfTest"; Microsoft notes:

Using CFR, features may be gradually rolled out, starting with devices that install the monthly optional non-security preview release. When we've validated that each feature is ready, we'll gradually roll it out to new devices, and eventually include it enabled-by-default in a subsequent monthly security update.

Two characteristics of the new routines immediately irritated me. The first irritation is the granularity of the captured packet timestamps; the documentation says:

TimeStamp – Timestamp when the packet was reported. This is retrieved using ‘KeQuerySystemTime’.

KeQuerySystemTime documentation includes the following remark:

System time is typically updated approximately every ten milliseconds.

My first test of the new routines was to capture packets and write them to a file in PCAPNG format. I was surprised, when looking at the packets in Wireshark, how many packets had identical timestamps (a lot of packets can be exchanged in ten to twenty milliseconds).

The second irritation is that the length of the packet is not included in the metadata describing a captured packet. The length of the captured data is, of course, available but if a TruncationSize (snapshot length) is specified, the original length of the packet is not available from the metadata (it might be possible to infer the original length of the packet from the packet contents). Since the original length and snapshot length are included in the PCAPNG packet block, I chose to capture full packets (TruncationSize = 9000).

Tracing HTTPS traffic on Microsoft Windows

Capturing HTTPS traffic is becoming an increasingly necessary troubleshooting technique (as HTTPS continues to replace plain HTTP), but is also becoming a more difficult undertaking. Assuming that one has control of the client end of the HTTPS channel, here are a few techniques that might be able to capture the traffic.

Network Sniffing

Because the network traffic is encrypted, a plain network trace will not show information about the HTTP protocol activity but it can nonetheless be interesting and/or useful.

Web browsers are keen users of experimental TCP mechanisms such as TCP Fast Open (TFO) and a network trace is useful for examining the initial TLS handshake steps – one can see which cipher suites are offered/accepted, the Server Name Indication (SNI) and Application-Layer Protocol Negotiation (ALPN) Client Hello extensions (if present) and the general shape/health of the TCP data flow.

Network Sniffing and Decryption (via Server Certificate Private Key)

The necessary ingredients for successfully capturing plaintext with this approach are: 

·         Access to the private key of the HTTPS server.

·         Ability to ensure that the client does not offer a cipher suite with ephemeral keys (“forward secrecy”).

·         Ability to ensure that TLS Session Resumption is not used.

·         Probability of capturing all relevant packets, otherwise the state information needed to generate Initialization Vectors (IV) for and verify message authentication codes (MAC) of subsequent TLS (Transport Layer Security) records may be lost.

The first condition can rarely be met; even if the authority responsible for the HTTPS server is willing, there may be technical obstacles to exporting the private key. The second condition is increasingly difficult to meet – HTTP/2 blacklists all cipher suites that do not use ephemeral keys.

Network Sniffing and Decryption (via Export Session Keys)

Some HTTPS clients offer the ability to export the TLS session keys (e.g. the SSLKEYLOGFILE setting for Chrome and Firefox browsers); this finesses the first 3 problems mentioned above. Some network trace analysis tools (such as Wireshark) can import the exported session keys and decrypt the captured data. The ability to use a network trace analysis tool is especially useful when HTTP/2 is in use because the binary encoding of HTTP/2 can easily be decoded and nicely presented by such tools.

Network Sniffing and Null Cipher Suite

The necessary ingredients for successfully capturing plaintext with this approach are: 

·         Ability to enable null cipher suites on the HTTPS server.

·         Ability to ensure that the client only offers null cipher suites.

The modifications to both server and client can be difficult (the null cipher suites are blacklisted by HTTP/2) and unless the problem being troubleshot is tied to HTTPS (such as token binding, TLS record encapsulation, etc.), it would be easier to just use a plain HTTP connection.

Debugging Proxy Server

Debugging Proxy Servers, such as Fiddler, are a common and general purpose method of capturing HTTPS traffic. FiddlerCore is included with Microsoft’s Message Analyzer and is the mechanism used when choosing the “Pre-encryption for HTTPS” scenario in that tool.

There are mechanisms that try to protect against “man-in-the-middle” interventions in HTTPS communications, such as “Public Key Pinning Extension for HTTP” (RFC 7469) and “Certificate Transparency” (RFC 6962). If an HTTPS client using these mechanisms cannot be configured to accept the proxy server certificate (hierarchy) then this technique cannot be used. There often is a way to configure additional certificates, since this is needed in the case of enterprises that mandate TLS interception proxies at their boundaries, but it needs to be found on a case by case basis.

Built-in Tracing in the Client

A major class of HTTPS clients, namely web browsers, often have built-in debugging and tracing facilities, intended for developers (Internet Explorer and Edge call them “(F12) Developer Tools”).

Unlike the previous techniques, these tools typically don’t provide a byte-by-byte record of HTTPS traffic because, for their typical audience, this information is too low-level – especially HTTP/2 binary encoded, framed and interleaved traffic.

Microsoft-Windows-WinINet and Microsoft-Windows-WinINet-Capture

WinINet (Windows Internet) is an API for accessing the Internet and it is used by both Edge and Internet Explorer, as well as many other applications. Two ETW (Event Tracing for Windows) providers give particular insight into the behaviour of the API: Microsoft-Windows-WinINet and Microsoft-Windows-WinINet-Capture.

Microsoft-Windows-WinINet-Capture is the simplest provider with just four events: the request/response headers/payloads. This “captures” all of the “data” exchanged, albeit that HTTP/2 data is mapped into an HTTP/1.1 style format (plain text rather than binary) and compressed content-encoding is expanded.

Microsoft-Windows-WinINet provides insight into the processing stages of an HTTP interaction and includes captured request/response headers and POST data. This provider also maps HTTP/2 binary encoded headers into HTTP/1.1 style plain text headers.

Microsoft-Windows-WebIO and Microsoft-Windows-WinHttp

WinHttp (Windows HTTP Services) is another API, similar to WinINet but intended for use in server/service scenarios. There are also two ETW providers associated with this API: Microsoft-Windows-WebIO and Microsoft-Windows-WinHttp.

Microsoft-Windows-WinHttp events are mostly related to proxy server discovery and use, and don’t give much insight into wider aspects of an HTTP interaction.

Microsoft-Windows-WebIO provides a similar level of detail to the WinINet provider. This provider mostly maps HTTP/2 binary encoded headers into HTTP/1.1 style plain text headers, but the sent headers are currently provided in some “intermediate” form (neither HTTP/2 binary encoded nor pure plain text).

Debugging of Schannel (Secure Channel) Interface

Intercepting the API calls that perform the encryption and decryption for TLS is another way of capturing the plain text of HTTPS communications. The WinINet, WinHttp and .NET Framework all use the Secure Channel (Schannel) security support provider via the Security Support Provider Interface (SSPI).

Tracing the input into EncryptMessage and the output from DecryptMessage captures all of the HTTPS content. One can also trace the input to and output from InitializeSecurityContext to capture the TLS connection establishment traffic.

.NET Framework .exe.config Tracing

The .NET Framework class library uses managed code to implement the HTTP/1.1 protocol and so its traffic is not observed by the ETW providers mentioned earlier (.NET Core does use the WinHttp API). There is however tracing built into the managed code implementation of HTTP that can be enabled and logged by appropriate settings in the application’s .config file.

Java Tracing

Java applications use Java implementations of the HTTP and TLS protocols. Like the .NET Framework, the Java implementation includes built-in debugging/tracing capabilities that can be enabled by setting the system property javax.net.debug.