Another approach is to “trace” the behaviour of the systems and either analyse the collected data or submit it for analysis by someone else. The network traffic and some of the events generated by Event Tracing for Windows providers are useful sources to trace.
Tracing network traffic
A straightforward network trace, performed on the client PC is a good first step. This could be performed with well-known tools (typically requiring installation) like Wireshark, Microsoft's Message Analyzer and Network Monitor or with built-in commands like “netsh trace” or the NetEventPacketCapture suite of PowerShell cmdlets (the well-known tools can also interpret and display captured network trace data).
One common class of problems is data sent by the VPN client that is never received by the VPN server. Ideally, one would perform a network trace on both the client and the server. It might be the case that fragments of the data reach the receiver but, perhaps because of a lossy network, a complete message is never received or it may be the case that the data is blocked/lost on route to the receiver. With just a network trace from one party, it is often possible to infer whether data is being blocked or lost but it is obviously better to compare the concurrent traces made by the two parties. If the problem can be easily reproduced, initially just tracing from the VPN client and only reverting to a concurrent client and server trace if the client-only trace is inconclusive is a pragmatic approach.
Tracing VPN protocol interactions
Most VPN protocols quickly switch from exchanging plaintext messages during initial connection set-up to exchanging encrypted messages. For the two IKE (Internet Key Exchange) based VPN protocols (L2TP/IPsec and IKEv2), tracing the Microsoft-Windows-RRAS, Microsoft-Windows-WFP and “IKEEXT Trace Provider” Event Tracing for Windows providers is most helpful.
In the format of a file suitable for use with the logman.exe “-pf” option, the following providers are a good starting point for the “Automatic” VPN type (where various VPN protocols are tried until one succeeds or they all fail):
All of these providers and a network trace as well can be started with a single command:
Trace Provider Types
There are several types of trace providers, including:
For the MOF, Manifest and TraceLogging types there is normally enough information in the trace output coupled with information included with the Windows installation to present all of the captured data in a human readable format.
For the WPP type, a “.pdb” (Program DataBase, debug symbols) file containing “TMF” or “PUBLIC_TMF” annotations is needed to interpret the trace data. “TMF” annotations are normally only available in private symbol files (i.e. symbols files only available to the developer (e.g. Microsoft)).
IKEEXT Trace Provider
The IKEEXT trace provider is a WPP provider type and difficult to interpret with the private symbols for ikeext.dll. Nonetheless, it is the most useful provider when one needs to understand IKE behaviour (for the L2TP/IPsec and IKEv2 VPN types).
For VPN connections that use a pre-shared key for authentication, the key will probably be present (in plaintext) in the trace data.
Microsoft-Windows-WFP
This is a manifest based provider for the Windows Filtering Platform. Most (if not all) of the useful information recorded by this provider is also present in an IKEEXT trace, but this trace data is more easily readable.
Microsoft-Windows-RRAS
This manifest based provider delivers useful information for all of the VPN types, as one would expect from the Routing and Remote Access Server provider.
Microsoft-Windows-Ras-NdisWanPacketCapture
This manifest based provider shows the packets that flow through the VPN in their unencrypted form. It is well suited to examining the initial data exchanges through the VPN (including DHCP) and subsequently recording which traffic uses the VPN (useful if there are any routing problems).
Microsoft-Windows-TCPIP
This manifest provider generates a lot of low-level TCP/UDP/IP information, including routing table changes and routing decisions. Because it is a high-volume source of events, if the size of the generated trace file might become an issue (e.g. when sharing the trace or even just analysing the trace) then this provider should only be enabled if there are good grounds to believe that it will be useful when analysing the problem.
Microsoft-Pef-WFP-MessageProvider and Microsoft-Windows-NDIS-PacketCapture
These manifest based providers record the raw network packets on the local network. They can be useful for the very first (unencrypted) message exchanges during the establishment of a VPN connection and diagnosing IP fragmentation/reassembly issues but since these providers cannot be simply started and stopped (Microsoft-Windows-NDIS-PacketCapture, for example, requires rebinding of network stacks), one should have good grounds for using them (similar to the Microsoft-Windows-TCPIP provider).
Microsoft-Windows-RasSstp
This manifest based provider is only useful for an SSTP VPN connection and probably logs at most one event per connection attempt – a description of an error that occurred.
Microsoft-Windows-WebIO
This manifest based provider is only useful for an SSTP VPN connection and is probably the most security sensitive provider: it traces WinHttp (API) activity from all process and records the HTTP headers (Including authentication headers) – if passwords can be extracted from the headers (e.g. in the case of Basic authentication) then the passwords are compromised.
Other Providers
There are many other event providers that can be useful under particular circumstances (RAS, VPN, EAP, NPS, IAS, MPR, BFE, Firewall, Security, etc. related). Typically, one would only search for and select such providers once one has gathered evidence that they might be useful.
General security considerations
Event tracing of VPN connections will probably reveal most of the VPN configuration data, including IP addresses and VPN user name – but not VPN passwords (as far as I am aware, unless PAP (Password Authentication Protocol) is used) or certificate private keys. As already mentioned, pre-shared keys could be compromised.
Many manifest providers that potentially generate events containing Personally Identifiable Information (PII) flags such events with a keyword like PII_PRESENT (often having a value of 0x20000000000). Examining the manifest (metadata) of a provider will show the keywords that it supports.
There is no practical method to remove security sensitive data from a trace file whilst preserving the integrity of the trace data.
Other event tracing considerations
If the rate of event generation is too high, events can be lost (i.e. not captured); there are tracing parameters that can be adjusted to increase the level of buffering (and therefore reduce the level of event loss). One should also try to keep the trace period short – reproducing the problem as quickly as possible.
The files created by event tracing can be sparsely populated (to assist performant logging) and can often be substantially compressed (perhaps to less than a tenth of the original size) – which can be useful if trace files are shared.
An understanding of VPN protocols is very useful when examining a trace file and a very high degree of protocol and Windows experience is needed to interpret VPN relevant WPP trace providers.
There are many event trace providers that can provide insights into specific aspects of VPN behaviour. It is not uncommon to repeatedly trace a VPN problem, modifying the selection of trace providers (and perhaps also tracing from both client and server) in response to the analysis of earlier traces.
Update 2023
Microsoft Message Analyzer has been retired and its associated packet capture driver (Microsoft-Pef-WFP-MessageProvider) no longer passes driver code signing checks.
PktMon became known to me. PktMon can (any, by default, does) trace both the VPN protocol packets and the encapsulated data carried by the VPN protocol (the same data as captured by Microsoft-Windows-Ras-NdisWanPacketCapture). The PktMon event provider Microsoft-Windows-PktMon requires the pktmon.sys driver to be started and configured via IOCTLs, so Microsoft-Windows-Ras-NdisWanPacketCapture is still useful when a “pure” ETW trace controller (such as WPR or logman) is being used to control tracing.
A minor bug was introduced into IKEEXT.dll, which I reported thus:
An exception can occur in IkeGetUdpEncapIfDoingNatt during L2TP or IKEv2 VPN connection establishment. A variable that is not initialized via all code paths is used in the WPP logging code at the end of the routine.
An effective workaround is to limit the trace level to level 4 (no particularly useful information is lost); I now use "IKEEXT Trace Provider" with keywords 0x10 and level 4.
A new TraceLogging provider named Microsoft.Windows.Networking.Ikeext
was introduced; this provider largely mirrors the same events as "IKEEXT
Trace Provider" but in an easier to decode format. The GUID for this
provider is consistent with the “ETW name-hashing algorithm”; for tools, such
as WPR which understand and support the convention, this provider can be specified
by preceding its name with a star/asterisk: *Microsoft.Windows.Networking.Ikeext
The IKEEXT trace providers "IKEEXT Trace Provider" and Microsoft.Windows.Networking.Ikeext can log plaintext versions of IKEv1 and IKEv2 packets ("IKEEXT Trace Provider" logs the full packet and Microsoft.Windows.Networking.Ikeext logs the first 511 bytes). The REG_DWORD registry value “TestFlags” under key “HKLM\SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters” enables this logging when set to the value 0x20. The key is monitored by IKEEXT for changes, so there is no need to restart IKEEXT after modifying the value.
The packets are logged as hex strings; when decoded to binary
and prepended with a dummy IP/UDP header, they can be viewed in tools such as Wireshark and Microsoft Message
Analyzer (if you still have a working copy). Received IKE packets can be
usefully displayed “as is” whereas sent IKE packets benefit from a few small “tweaks”
before display (setting the Length field in the IKE header, reversing the bytes
of the Message ID in the IKE header, setting the Payload length of the IKEv2 Encrypted
payload to cover just the length of the initialisation vector, clearing the
encryption flag in an IKEv1 header).
