Analyzing Windows heap usage with and without ETW

 

It has been a long time since I last wanted to discover if/where a program was “leaking” heap allocations. Most programs that I developed myself just performed some task and exited; heap allocations (from all sources, including Microsoft and other third party DLLs) probably rarely exceeded a few megabytes. I coded mostly with C# (garbage collected); most heap allocations directly under my control arose from native interop and I adopted an approach of releasing memory when it was “easy” and did not obscure the main intent of the code – otherwise I “intentionally” allowed the memory to leak.

I mention the above because I am a heavy user of Event Tracing for Windows (ETW) but I had hitherto no experience of using ETW (or, indeed, any other tool) to investigate heap usage. It was only when I tried to help with a problem/question in a technical forum that I had a need to understand heap usage. The question was whether the Windows Filtering Platform API FwpmNetEventEnum unavoidably leaks heap allocations.

The first approach that came to mind was to use the User-Mode Dump Heap (UMDH) utility from the Debugging Tools for Windows kit. However, the “current” version did not seem to work. Searching the web for explanations uncovered the following quotes for other users who had encountered the problem:

According to a Microsoft employee, this is a known problem. I quote: "Yeah. It's not working and I don't know when/if it will ever be."

I also quote an email I got from a Microsoft Support guy: "Anyway, I have confirmation it is broken. The dev team owning the exe knows about it and when they can get to fixing it they will."

Fortunately older versions of UMDH still work and it quickly became apparent that FwpmNetEventEnum does leak heap allocations. Most Fwpm* routines use RPC to the Base Filtering Engine (BFE) service to perform their function. Those Fwpm* APIs that return complex data structures mostly use a [allocate(all_nodes)] attribute in the MIDL ACF (Application Configuration File) so that the data can be freed with a single call to midl_user_free; however, that attribute was not applied to the RPC routine at the core of FwpmNetEventEnum. A subsequent call to FwpmFreeMemory just frees the top-level allocation and not the additional embedded allocations.

The absence of the [allocate(all_nodes)] attribute could be confirmed with tools that dump embedded RPC data structures; one example of a heap allocation back-trace that demonstrated that complex data structures were being allocated node-by-node was:

ntdll!RtlpAllocateHeapInternal+0x80B4E

fwpuclnt!MIDL_user_allocate+0x19

RPCRT4!NdrSafeAllocate+0x47

RPCRT4!Ndr64ComplexStructUnmarshall+0x72D

RPCRT4!Ndr64EmbeddedPointerUnmarshall+0x366

RPCRT4!Ndr64UnionUnmarshall+0x2D9

RPCRT4!Ndr64ComplexStructUnmarshall+0x5F4

RPCRT4!Ndr64pPointerLayoutUnmarshallCallback+0x234

RPCRT4!Ndr64ConformantArrayUnmarshall+0x21C

RPCRT4!Ndr64TopLevelPointerUnmarshall+0x40F

RPCRT4!Ndr64TopLevelPointerUnmarshall+0x59D

RPCRT4!Ndr64pClientUnMarshal+0x2A1

RPCRT4!NdrpClientCall3+0x40C

RPCRT4!NdrClientCall3+0xEB

fwpuclnt!FwpmNetEventEnum5+0x70

Heap Snapshots

I then turned my thoughts to understanding what type of bug could have been introduced into UMDH. There are several methods of obtaining the information needed to dump heap snapshot information (including heap allocation back-traces) about a process; the routines RtlQueryProcessDebugInformation and RtlQueryHeapInformation can both independently obtain the necessary information. UMDH seems to have taken a different approach and used the routine ReadProcessMemory and a knowledge of NTDLL internal data structures to gather the information.

The failing version of UMDH seems to have started using RtlQueryHeapInformation (with an HEAP_INFORMATION_CLASS value of HeapExtendedInformation (2)) to obtain information about heap allocations, but this information does not include any data that can be used to associate the allocation with a back-trace. There is, however, a HEAP_INFORMATION_CLASS value (5, let’s name it HeapStackTraceInformation) that returns information well suited for use by UMDH (i.e. includes information about allocated heap blocks and back-traces for the allocations).

The back-traces returned by RtlQueryHeapInformation for HeapStackTraceInformation come from a different source compared to the back-traces created and store when the Global Flag FLG_USER_STACK_TRACE_DB is set. The back-traces used by RtlQueryHeapInformation are enabled and disabled by RtlSetHeapInformation (also with a HEAP_INFORMATION_CLASS value of 5) or by creating a value named “FrontEndHeapDebugOptions” under the Image File Execution Options (IFEO) key for an image; this value can be set by the Windows Performance Recorder (WPR) command “wpr -snapshotconfig heap –name […]” (“wpr -snapshotconfig heap –pid […]” effectively calls RtlSetHeapInformation).

When comparing the two versions of the back-trace information for a given allocation, they mostly just differ in the first frame:

HeapStackTraceInformation:

ntdll!RtlpAllocateHeapInternal+0x80b49:

e8528d0500      call    ntdll!RtlpHpStackTraceAddStack

 

FLG_USER_STACK_TRACE_DB:

ntdll!RtlpAllocateHeapInternal+0x809dd:

e8ac1cffff      call    ntdll!RtlpCallInterceptRoutine

The back-traces can also differ in the depth of the back-trace captured and stored (HeapStackTraceInformation can save more frames).

“wpr -singlesnapshot heap […]” uses EnableTraceEx2 to send an EVENT_CONTROL_CODE_CAPTURE_STATE to the Microsoft-Windows-Heap-Snapshot provider, using the EnableFilterDesc field of the EnableParameters parameter to select the “pids”. This causes RtlQueryHeapInformation with HeapStackTraceInformation to be executed in the target processes with the output being broken into chunks and logged into the trace session. Windows Performance Analyzer (WPA) can reassemble, analyze and display this data in a “Heap Snapshot” graph.

Heap Events

WPR provides another heap related command: “wpr -heaptracingconfig […]”. This command creates/sets another value under IFEO – namely TracingFlags. These flags enable aspects of the User Mode Global Logger (UMGL), including events generated by the WMI HeapTraceProvider; this provider generates events for individual heap events (HeapRangeCreate, HeapRangeReserve, HeapRangeRelease, HeapRangeDestroy, HeapCreate, HeapAllocation, HeapReallocation, HeapDestroy, HeapFree and more) and StackWalk back-traces can be configured for selected event types. WPA knows how to analyze and display these events too (in various graphs in the Memory category).

The instrumentation for these events is obviously embedded in many NTDLL heap routines; for the HeapAllocation event, the instrumentation is embedded close to the heap stack tracing calls:

ntdll!RtlpAllocateHeapInternal+0x80aec:

e817a30500      call    ntdll!RtlpLogHeapAllocateEvent

If a process was started without heap tracing enabled via IFEO, heap tracing can still be enabled by directly setting the heap tracing bit in the _PEB.TracingFlags field (perhaps via a debugger); there does not seem to be any API that performs this function.

Exploring Identity Privacy, TEAP and TTLS in conjunction with Microsoft Network Policy Server

 

Windows 10/11 clients include at least three EAP (Extensible Authentication Protocol) methods than can both be used in conjunction with 802.1X or VPN protocols and also support identity privacy/protection: PEAP (Protected EAP), TEAP (Tunnel EAP) and EAP-TTLS (EAP Tunneled Transport Layer Security).

On the server side, NPS (Network Policy Server) only supports PEAP natively but additional EAP methods can be plugged in (either as legacy methods or EAPHost based methods).

Enabling identity privacy on the client side is straightforward: just enable the feature and optionally specify a name to be used as the outer (visible) identity. How to “enable” identity privacy on the server side is less obvious; I have used quotation marks around “enable” because identity privacy is not explicitly enabled – one just has to ensure that the NPS configuration is compatible with identity privacy.

NPS has two types of policies: Connection Request Policies (CRP) and Network Policies (previously known as Remote Access Policies (RAP)). The policy type names don’t clearly demarcate their intended purposes. The Microsoft documentation says:

Connection request policies are sets of conditions and settings that allow network administrators to designate which Remote Authentication Dial-In User Service (RADIUS) servers perform the authentication and authorization of connection requests that the server running Network Policy Server (NPS) receives from RADIUS clients. Connection request policies can be configured to designate which RADIUS servers are used for RADIUS accounting.

Network policies are sets of conditions, constraints, and settings that allow you to designate who is authorized to connect to the network and the circumstances under which they can or cannot connect.

Configuring authentication (allowed authentication mechanisms, etc.) seems like something that should be part of the network policy and this indeed is where it is typically defined. However, when searching Microsoft documentation for help on configuring identity privacy, only this short “tip” is readily findable:

Tip

The NPS policy for 802.1X Wireless must be created by using NPS Connection Request Policy. If the NPS policy is created in by using NPS Network Policy, then identity privacy will not work.

The implications of this simple statement are also not immediately obvious. Awareness of the availability of a setting named “Override network policy authentication settings” suggests a possibility:

[…] if the option to Override network policy authentication settings is enabled on the Settings tab in a connection request policy, then authentication is performed in connection request policy. Otherwise, authentication is performed in network policy. Authentication can be configured in both types of policies.

RADIUS requests to NPS are processed by a “pipeline” of stages, defined at HKLM\SYSTEM\CurrentControlSet\Services\RemoteAccess\Policy\Pipeline; the stages (in order of evaluation) are:

	Stage	Providers	Reasons	Replays	Requests	Responses
1	IAS.ProxyPolicyEnforcer				0 1 2	0 1 2 3 4
2	IAS.Realm	1			0 1	0
3	IAS.Realm	0 2			0 1	0
4	IAS.NTSamNames	1			0	0
5	IAS.CRPBasedEAP	1			0 2	0
6	IAS.Realm	1		0	0	0
7	IAS.NTSamNames	1		0	0	0
8	IAS.MachineNameMapper	1		0	0	0
9	IAS.BaseCampHost			0
10	IAS.RadiusProxy	2		0		0
11	IAS.ExternalAuthNames	2		0		0
12	IAS.NTSamAuthentication	1		0	0	0 1 2
13	IAS.UserAccountValidation	1 3	33	0	0	0 1
14	IAS.MachineAccountValidation	1		0	0	0 1
15	IAS.EAPIdentity	1		0	0	0 1
17	IAS.PolicyEnforcer	1 3	33	0	0	0 1
18	IAS.NTSamPerUser	1 3	33	0	0	0 1
19	IAS.URHandler	1 3	33	0	0	0 1
20	IAS.RAPBasedEAP	1		0	0 2	0
21	IAS.PostEapRestrictions	0 1 3		0	0	0 1
23	IAS.ChangePassword	1		0	0	1
24	IAS.AuthorizationHost			0
25	IAS.EAPTerminator	0 1		0	0 2	1 2 3 5
26	IAS.DatabaseAccounting
27	IAS.Accounting
28	IAS.MSChapErrorReporter	0 1 3		0	0	2

Providers: 0 → None, 1 → Windows, 2 → RADIUS Proxy, 3 → External Authentication

Reasons: 33 → PASSWORD_MUST_CHANGE

Replays: 0 → FALSE

Requests: 0 → ACCESS_REQUEST, 1 → ACCOUNTING, 2 → CHALLENGE_RESPONSE

Responses: 0 → INVALID, 1 → ACCESS_ACCEPT, 2 → ACCESS_REJECT, 3 → ACCESS_CHALLENGE, 4 → ACCOUNTING, 5 → DISCARD_PACKET

Each stage gets a chance to handle the request, if the current request state (provider, reason, replays, requests, responses) allows.

If the “outer” identity does not exist, stage 13 (IAS.UserAccountValidation) reports reason NO_SUCH_USER.

If the “outer” identity exists but is disabled, stage 13 (IAS.UserAccountValidation) reports reason ACCOUNT_EXPIRED.

If the “outer” identity is usable but differs from the “inner” identity, stage 20 (IAS.RAPBasedEAP) reports problem ERROR_PEAP_IDENTITY_MISMATCH.

If the authentication settings are set on the Connection Request Policy then stage 5 (IAS.CRPBasedEAP) gets a chance to handle the request and influence its handling by later pipeline stages.

The NPS log file entries for a PEAP-MSCHAPv2 authenticated VPN session with identity privacy contain the following usernames:

User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "anonymous"
User-Name = "GARY"
User-Name = "GARY"
User-Name = "GARY"
User-Name = "GARY"
User-Name = "anonymous"
User-Name = "anonymous"

There are 24 entries in total (the “challenge response” entries don’t have a username value). The initial EAP and EAP TLS establishment entries contain the “outer” identity, the inner MSCHAPv2 exchanges contain the “inner” identity and the accounting start/stop entries use the “outer” identity.

Implementing “toy” server-side (authenticator) TEAP and EAP-TTLS support

Each client-side (supplicant) EAP method includes a “properties” value in the registry; below is a summary of five of the methods installed by default in Windows 10/11.

Property	Value	TEAP	PEAP	EAP-TTLS	EAP-TLS	MSCHAPv2
PropCipherSuiteNegotiation	0x00000001	X	X	X	X
PropMutualAuth	0x00000002	X	X	X	X	X
PropIntegrity	0x00000004	X	X	X	X	X
PropReplayProtection	0x00000008	X	X	X	X	X
PropConfidentiality	0x00000010	X	X
PropKeyDerivation	0x00000020	X	X	X	X	X
PropKeyStrength64	0x00000040					X
PropKeyStrength128	0x00000080	X	X	X	X
PropKeyStrength256	0x00000100
PropKeyStrength512	0x00000200
PropKeyStrength1024	0x00000400
PropDictionaryAttackResistance	0x00000800	X	X	X	X
PropFastReconnect	0x00001000	X	X	X	X
PropCryptoBinding	0x00002000	X	X
PropSessionIndependence	0x00004000	X	X	X	X	X
PropFragmentation	0x00008000	X	X	X	X
PropChannelBinding	0x00010000
PropNap	0x00020000		X
PropStandalone	0x00040000	X	X	X		X
PropMppeEncryption	0x00080000	X	X	X	X	X
PropTunnelMethod	0x00100000	X	X	X
PropSupportsConfig	0x00200000	X	X	X	X	X
PropCertifiedMethod	0x00400000	X
PropHiddenMethod	0x00800000
PropMachineAuth	0x01000000	X	X	X	X	X
PropUserAuth	0x02000000	X	X	X	X	X
PropIdentityPrivacy	0x04000000	X	X	X
PropMethodChaining	0x08000000	X
PropSharedStateEquivalence	0x10000000	X	X	X	X
	0x20000000	X
PropReserved	0x80000000

Based on experience analyzing PEAP behaviour, I thought that it would be feasible to implement server-side (authenticator) support for EAP-TTLS and TEAP; furthermore, I expected a good deal of commonality between the implementations, so I created a generic (in the C# sense) tunnel EAP class to handle common tasks (such as establishing the TLS tunnel using Schannel, EAP identity, EAP negotiation, EAP fragmentation) which could be specialized by types that could handle method specific tasks (such as TLV or AVP encapsulation, master session key generation, etc.).

There are two development frameworks for EAP methods: legacy EAP and EAPHost. Because the implementation would be tunneling EAP-MSCHAPv2 (which is a legacy EAP implementation) and because PEAP is also a legacy EAP implementation, I chose to use the legacy EAP framework too.

Implementing EAP-TTLS without support for identity privacy was straightforward. It (more specifically EAP-TTLSv0) does not support cryptobinding, so all that is required is packing and unpacking EAP messages in TTLS AVP (attribute-value pair) format and calculating/obtaining the master session key (available via SetContextAttributes /QueryContextAttributes with SecPkgContext_EapPrfInfo/SecPkgContext_EapKeyBlock and “EAP-TTLSv0 Keying Material”).

Adding identity privacy support involves use of undocumented (or poorly documented) features of the Windows EAP frameworks. In particular, the “action” EAPACTION_IndicateIdentity (documentation: “Reserved for system use”) or its EAPHost equivalent EAP_METHOD_AUTHENTICATOR_RESPONSE_AUTHENTICATE (documentation: “The authenticator method has started authentication of the supplicant” – at best unhelpful and possibly totally incorrect). It seems that although frameworks for new EAP methods are supported, it was not expected that new “tunnel” EAP methods would be needed.

The EAPACTION_IndicateIdentity action allows NPS to be informed of the inner identity which can then be passed on to NPS policy stages such as IAS.UserAccountValidation that might otherwise fail the authentication on the basis of the outer identity.

TEAP

The TEAP RFC is twice the length of the EAP-TTLS RFC (100 pages vs. 50 pages) and there is also quite a lengthy “Errata” report for the TEAP RFC. Before reading the errata, I puzzled over whether an Intermediate-Result TLV was necessary if there was only one inner method: the RFC is ambiguous (with a tendency to “not necessary”), the errata says that an Intermediate-Result TLV is mandatory and the Windows TEAP client expects such a TLV. I also make a mistake with the “MSK Compound MAC” in the crypto-binding TLV (not truncating the value when, for example, SHA-384 is used instead of SHA-1).

Fortunately the ETW tracing for TEAP (provider Microsoft.Windows.Eap.Teap) is very helpful. This highlighted extract shows some of the useful information in the trace data:

The keying information from the TLS tunnel that is needed for the TEAP cryptographic calculations can be obtained via SetContextAttributes /QueryContextAttributes with SecPkgContext_KeyingMaterialInfo/SecPkgContext_KeyingMaterial.