Tracing HTTPS traffic on Microsoft Windows

Capturing HTTPS traffic is becoming an increasingly necessary troubleshooting technique (as HTTPS continues to replace plain HTTP), but is also becoming a more difficult undertaking. Assuming that one has control of the client end of the HTTPS channel, here are a few techniques that might be able to capture the traffic.

Network Sniffing

Because the network traffic is encrypted, a plain network trace will not show information about the HTTP protocol activity but it can nonetheless be interesting and/or useful.

Web browsers are keen users of experimental TCP mechanisms such as TCP Fast Open (TFO) and a network trace is useful for examining the initial TLS handshake steps – one can see which cipher suites are offered/accepted, the Server Name Indication (SNI) and Application-Layer Protocol Negotiation (ALPN) Client Hello extensions (if present) and the general shape/health of the TCP data flow.

Network Sniffing and Decryption (via Server Certificate Private Key)

The necessary ingredients for successfully capturing plaintext with this approach are: 

·         Access to the private key of the HTTPS server.

·         Ability to ensure that the client does not offer a cipher suite with ephemeral keys (“forward secrecy”).

·         Ability to ensure that TLS Session Resumption is not used.

·         Probability of capturing all relevant packets, otherwise the state information needed to generate Initialization Vectors (IV) for and verify message authentication codes (MAC) of subsequent TLS (Transport Layer Security) records may be lost.

The first condition can rarely be met; even if the authority responsible for the HTTPS server is willing, there may be technical obstacles to exporting the private key. The second condition is increasingly difficult to meet – HTTP/2 blacklists all cipher suites that do not use ephemeral keys.

Network Sniffing and Decryption (via Export Session Keys)

Some HTTPS clients offer the ability to export the TLS session keys (e.g. the SSLKEYLOGFILE setting for Chrome and Firefox browsers); this finesses the first 3 problems mentioned above. Some network trace analysis tools (such as Wireshark) can import the exported session keys and decrypt the captured data. The ability to use a network trace analysis tool is especially useful when HTTP/2 is in use because the binary encoding of HTTP/2 can easily be decoded and nicely presented by such tools.

Network Sniffing and Null Cipher Suite

The necessary ingredients for successfully capturing plaintext with this approach are: 

·         Ability to enable null cipher suites on the HTTPS server.

·         Ability to ensure that the client only offers null cipher suites.

The modifications to both server and client can be difficult (the null cipher suites are blacklisted by HTTP/2) and unless the problem being troubleshot is tied to HTTPS (such as token binding, TLS record encapsulation, etc.), it would be easier to just use a plain HTTP connection.

Debugging Proxy Server

Debugging Proxy Servers, such as Fiddler, are a common and general purpose method of capturing HTTPS traffic. FiddlerCore is included with Microsoft’s Message Analyzer and is the mechanism used when choosing the “Pre-encryption for HTTPS” scenario in that tool.

There are mechanisms that try to protect against “man-in-the-middle” interventions in HTTPS communications, such as “Public Key Pinning Extension for HTTP” (RFC 7469) and “Certificate Transparency” (RFC 6962). If an HTTPS client using these mechanisms cannot be configured to accept the proxy server certificate (hierarchy) then this technique cannot be used. There often is a way to configure additional certificates, since this is needed in the case of enterprises that mandate TLS interception proxies at their boundaries, but it needs to be found on a case by case basis.

Built-in Tracing in the Client

A major class of HTTPS clients, namely web browsers, often have built-in debugging and tracing facilities, intended for developers (Internet Explorer and Edge call them “(F12) Developer Tools”).

Unlike the previous techniques, these tools typically don’t provide a byte-by-byte record of HTTPS traffic because, for their typical audience, this information is too low-level – especially HTTP/2 binary encoded, framed and interleaved traffic.

Microsoft-Windows-WinINet and Microsoft-Windows-WinINet-Capture

WinINet (Windows Internet) is an API for accessing the Internet and it is used by both Edge and Internet Explorer, as well as many other applications. Two ETW (Event Tracing for Windows) providers give particular insight into the behaviour of the API: Microsoft-Windows-WinINet and Microsoft-Windows-WinINet-Capture.

Microsoft-Windows-WinINet-Capture is the simplest provider with just four events: the request/response headers/payloads. This “captures” all of the “data” exchanged, albeit that HTTP/2 data is mapped into an HTTP/1.1 style format (plain text rather than binary) and compressed content-encoding is expanded.

Microsoft-Windows-WinINet provides insight into the processing stages of an HTTP interaction and includes captured request/response headers and POST data. This provider also maps HTTP/2 binary encoded headers into HTTP/1.1 style plain text headers.

Microsoft-Windows-WebIO and Microsoft-Windows-WinHttp

WinHttp (Windows HTTP Services) is another API, similar to WinINet but intended for use in server/service scenarios. There are also two ETW providers associated with this API: Microsoft-Windows-WebIO and Microsoft-Windows-WinHttp.

Microsoft-Windows-WinHttp events are mostly related to proxy server discovery and use, and don’t give much insight into wider aspects of an HTTP interaction.

Microsoft-Windows-WebIO provides a similar level of detail to the WinINet provider. This provider mostly maps HTTP/2 binary encoded headers into HTTP/1.1 style plain text headers, but the sent headers are currently provided in some “intermediate” form (neither HTTP/2 binary encoded nor pure plain text).

Debugging of Schannel (Secure Channel) Interface

Intercepting the API calls that perform the encryption and decryption for TLS is another way of capturing the plain text of HTTPS communications. The WinINet, WinHttp and .NET Framework all use the Secure Channel (Schannel) security support provider via the Security Support Provider Interface (SSPI).

Tracing the input into EncryptMessage and the output from DecryptMessage captures all of the HTTPS content. One can also trace the input to and output from InitializeSecurityContext to capture the TLS connection establishment traffic.

.NET Framework .exe.config Tracing

The .NET Framework class library uses managed code to implement the HTTP/1.1 protocol and so its traffic is not observed by the ETW providers mentioned earlier (.NET Core does use the WinHttp API). There is however tracing built into the managed code implementation of HTTP that can be enabled and logged by appropriate settings in the application’s .config file.

Java Tracing

Java applications use Java implementations of the HTTP and TLS protocols. Like the .NET Framework, the Java implementation includes built-in debugging/tracing capabilities that can be enabled by setting the system property javax.net.debug.