Establishing a VPN connection from macOS to Windows 10

Windows 10 (i.e. a non-server version of Windows) can act as a VPN server. Searching the Internet for “Windows 10” coupled with “VPN server” or “incoming connection” will probably turn up a few guides on how to set up Windows 10 as a VPN server. However, the difficulties of matching VPN client and server capabilities in terms of VPN transport and authentication protocols are often not mentioned. This article looks at the concrete case of establishing a VPN connection from a macOS client to a Windows 10 server using built-in VPN functionality (i.e. not using third party VPN products on either system).

The version of macOS (High Sierra, 10.13.6) used for testing purposes offers 3 “types” of VPN:

1.       L2TP over IPSec

2.       Cisco IPSec

3.       IKEv2

Windows 10 offers 4 types:

1.       PPTP

2.       L2TP/IPsec

3.       SSTP

4.       IKEv2

Cisco IPsec uses “Extended Authentication within IKE (XAUTH)” (for which there is a draft RFC from 2001: draft-beaulieu-ike-xauth-02.txt). Windows rejects connection attempts that try to negotiate these extensions.

SSL VPN (VPN over SSL) clients are available for macOS from the Apple Store, but a cursory examination did not find any that claim compatibility with Microsoft SSTP.

Using L2TP over IPsec

The transport protocol (L2TP in an IPsec Encapsulating Security Payload (ESP) tunnel) is common to both systems (Windows 10 and macOS).

There are two macOS options for “Machine Authentication” (IKEv1 Phase 1 authentication):

1.       Shared Secret

2.       Certificate

Shared Secret

“Shared Secret” seems easiest for a simple home network set-up, but the Windows “Network Connections” control panel applet does not provide any “user interface” for setting a shared secret (this applies to the user interfaces for the VPN server). There is however a trick that seems to work: define a “dummy” IPsec tunnel with “Preshared key” authentication; any tool can be used to do this (e.g. the “Windows Defender Firewall with Advanced Security” control panel applet, the “netsh advfirewall consec add rule” command or the “New-NetIPsecRule” PowerShell cmdlet).

Here are examples of creating suitable “dummy” IPsec tunnels (the endpoint value can be any address that does not apply to any potential traffic) with a shared secret of “XXX”:

netsh advfirewall consec add rule name="Dummy" endpoint1=192.168.3.1/32 endpoint2=any mode=tunnel action=requireinrequireout auth1=computerpsk auth1psk="XXX"

New-NetIPsecRule -DisplayName "Dummy" -Phase1AuthSet (New-NetIPsecPhase1AuthSet -DisplayName "Dummy" -Proposal (New-NetIPsecAuthProposal -Machine -PreSharedKey "XXX")).InstanceID -InboundSecurity Require -OutboundSecurity Require -KeyModule IKEv1 -Mode Tunnel -LocalAddress 192.168.3.1/32 -RemoteAddress Any

Obviously, having more than one IPsec tunnel definition with a (different) pre-shared key is problematic.

Certificate

The “Certificate” option is also not straightforward. By default, the registry value of “ServerFlags” (documented in section 2.2.3.4.6 of [MS-RRASM]: Routing and Remote Access Server (RRAS) Management Protocol) under the key “HKLM\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters” does not include the option “Authentication using certificates is allowed on the RRAS server” (0x04000000), so this must be explicitly added.

One also needs to create and manage a small certificate authority. I initially made a mistake by omitting the text highlighted below when creating a “root authority” certificate:

New-SelfSignedCertificate -CertStoreLocation Cert:\CurrentUser\My -KeyAlgorithm RSA -KeyLength 4096 -KeyExportPolicy Exportable -KeyUsage CertSign -NotAfter 2061-08-01 -Subject "CN=insert your name here" -Type Custom -TextExtension "2.5.29.19={text}CA=True"

The absence of the highlighted text did not affect the efficacy of the “root” certificate under Windows but caused “Trust evaluate failure: [root AnchorTrusted BasicConstraints]” under macOS.

When configuring the L2TP VPN under macOS, there is a field named “Server Address”; for the “Shared Secret” option, a dotted notation IPv4 address can be used but for the “Certificate” option one has to use a name (matching the certificate’s subject common name or DNS subject alternative name) and add the name and address to /etc/hosts.

Phase 2 (User) Authentication

macOS offers 5 possibilities (Password, RSA SecurID, Certificate, Kerberos, CryptoCard), only one of which I bothered to pursue: “Password” – this choice results in MSCHAPv2 authentication taking place.

Summary

L2TP was documented as RFC 2661 in 1999 – decidedly older than the other macOS option of IKEv2 (which was documented as RFC 5996 in 2010 and obsoleted by RFC 7296 in 2014). The transport protocol suffers from quite a high degree of encapsulation – below is a view (taken from Microsoft’s Message Analyzer) after the IPsec ESP encapsulation has been removed (replaced by “IP in IP” encapsulation after decryption – shown as Message2V4 by Message Analyzer):

Using IKEv2

When configuring the IKEv2 VPN under macOS, there are fields named “Server Address” and “Remote ID”; in contrast to the L2TP VPN, one has to use a dotted notation IPv4 address for the “Server Address” because it seemed as though macOS only tries to resolve the name via DNS (and not via /etc/hosts). The “Remote ID” field needs a name matching the VPN server certificate.

The “Authentication Settings” for IKEv2 are structured differently from L2TP; macOS offers: Username, Certificate, None.

The name “None” is rather misleading – one can interpret it as “do not use EAP” (i.e. include an AUTH payload from the first message in the IKE_AUTH exchange – the absence of such a payload indicates that EAP will be used instead). In this case both server and client must possess and use certificates to authenticate. The difficulties of using EAP (coding a replacement for a component not included in Windows 10) makes “None” a good choice of authentication mechanism.

EAP

“Certificate” authentication means authenticate with EAP-TLS and “Username” authentication means authenticate with EAP-MSCHAPv2 or PEAP.

Both authentication methods need to use EAP, but this is also disabled by default in the Windows RemoteAccess “ServerFlags” setting. One needs to explicitly set “EAP protocol can be negotiated for remote access and demand dial connection authentication” (0x00008000).

The file %SystemRoot%\System32\ias\ias.xml seems to be a replacement for the policy database of a full NPS (Network Policy Server) and some changes need to be made there too. In the section “Connections_to_Microsoft_Routing_and_Remote_Access_server”, the collection of “msNPAuthenticationType2” elements needs to be expanded to include “5” (IAS_AUTH_EAP) and optionally “11” (IAS_AUTH_PEAP); if IAS_AUTH_PEAP is added (and one wants to use PEAP) then the collection of “msNPAllowedEapType” elements needs to be expanded to include “19000000000000000000000000000000”.

Missing EAP Component

Windows 10 does not include the “Microsoft EAPHost Authenticator service” DLL (eapahost.dll) although it does include many (all?) other EAP components found on Windows Server. Fortunately, this DLL seems to have a simple task: to isolate the Windows VPN components from custom EAP implementations and to unify the two types of custom EAP implementations: legacy EAP methods (HKLM\SYSTEM\CurrentControlSet\Services\RasMan\PPP\EAP) and EapHost methods (HKLM\SYSTEM\CurrentControlSet\Services\Eaphost\Methods).

One needs to implement the interface below (described in a separate article) and package the result as a COM out-of-process server:

[ComImport, InterfaceType(ComInterfaceType.InterfaceIsIUnknown), Guid("5A8371A3-0C6D-487B-B3C8-46D785C4C940")]

interface IEapHostAuthenticatorSessionApis

{

    void BeginSession(uint flags, EAP_HOST_AUTHENTICATOR_METHOD_DATA_ARRAY methods, EAP_ATTRIBUTES ea, uint maxpack, [MarshalAs(UnmanagedType.LPWStr)] string identity, out uint session, out EAP_ERROR error);

    void UpdateInnerMethodParams(uint session, uint flags, [MarshalAs(UnmanagedType.LPWStr)] string identity, EAP_ATTRIBUTES ea, out EAP_ERROR error);

    void IsReplay(uint session,  uint n, EapPacket packet, [MarshalAs(UnmanagedType.Bool)] out bool replay, out EAP_ERROR error);

    void ReceivePacket(uint session, uint n, IntPtr p, out EAP_METHOD_AUTHENTICATOR_RESPONSE_ACTION action, out EAP_ERROR error);

    void SendPacket(uint session, out uint n, out IntPtr p, out EAP_AUTHENTICATOR_SEND_TIMEOUT timeout, out EAP_ERROR error);

    void GetAttributes(uint session, [Out] EAP_ATTRIBUTES ea, out EAP_ERROR error);

    void SetAttributes(uint session, EAP_ATTRIBUTES ea, out EAP_METHOD_AUTHENTICATOR_RESPONSE_ACTION action, out EAP_ERROR error);

    void SetAuthenticateResult(uint session, uint status, [Out] EAP_ATTRIBUTES ea, out EAP_METHOD_AUTHENTICATOR_RESPONSE_ACTION action, out EAP_ERROR error);

    void GetResult(uint session, [Out] EAP_METHOD_AUTHENTICATOR_RESULT2 result, out EAP_ERROR error);

    void GetFinalPacket(uint session, [MarshalAs(UnmanagedType.Bool)] bool success, out uint n, out IntPtr p, out EAP_ERROR error);

    void EndSession(uint session, out EAP_ERROR error);

}

A “proof-of-concept” implementation (including all necessary C# structure and enumeration definitions) occupied about 500 lines of code (quite small), but it violates the “using only built-in functionality” goal. Using the “None” authentication method is the way to meet that goal.

IKEv2 Policy Settings

The default policy settings are documented in section 2.2.3.4.2.8 of “[MS-RRASM]: Routing and Remote Access Server (RRAS) Management Protocol”. In summary, they are:

IntegrityMethod	INTEGRITY_SHA_256
EncryptionMethod	CIPHER_AES_256
CipherTransformConstant	CIPHER_CONFIG_CBC_3DES
AuthTransformConstant	AUTH_CONFIG_HMAC_SHA_256_128
PfsGroup	PFS_2048
DHGroup	DH_GROUP_2

The use of Triple DES for the Quick Mode (QM) cipher might be considered weak – as might the use of Diffie-Hellman (DH) Group 2. With these defaults, macOS uses DH Group 14 in its first IKE_SA_INIT exchange, only to have that rejected when Windows replies with a Notify payload of type INVALID_KE_PAYLOAD and macOS falls back to DH Group 2. A stronger policy can be configured, using the information in [MS-RRASM].

Summary

The view below (taken from Microsoft’s Message Analyzer) after IPsec ESP encapsulation has been removed shows how little encapsulation takes place compared to an L2TP VPN:

Addressing, Routing and Firewall

The only interesting setting option on the Windows 10 “Incoming Connections” object is the “IP address assignment”. The option “Assign IP addresses automatically using DHCP” does work, even if there is no DHCP service – if there is no DHCP service, the VPN server allocates addresses itself from the 169.xxx.xxx.xxx range; however a VPN connection attempt might fail with ERROR_IPSEC_IKE_INNER_IP_ASSIGNMENT_FAILURE before it is determined that no DHCP service is available. I chose “Specify IP addresses” and used a sub-range of the local network.

Although Windows 10 will forward IP traffic, the Windows 10 VPN server does nothing to advertise routes. The Windows 10 VPN server will however respond appropriately to ARP requests for its VPN clients.

By default, the VPN network will be assigned to the “Public” firewall profile (which, by default, blocks access to many services). After changing the profile to “Private”, packets were still dropped by a blocking WFP filter with a FWPM_CONDITION_ORIGINAL_PROFILE_ID condition unless the requested service was accessible via the “Public” profile – this is just an observation (the cause and resolution have not yet been determined).