This article describes the solution for the “access denied” error occurs when using qwinsta.exe/rwinsta.exe or WTSOpenServer Windows API function to access to a Windows XP SP2 PC.

The same thing happens when you are trying to connect to a Windows XP SP2 using Terminal Services Manager from a Windows 2003 Server. 

- Because essentially qwinsta.exe command calls WTS functions (WTSOpenServer, WTSEnumerateSessions, …), you will encounter the same error when using either the command or the API.

As a quick answer, check the following points.
1. Ensure you have the correct credential
Although you do not need to provide a username and a password when executing qwinsta.exe or WTSOpenServer, Windows use its stored credentials automatically, just like accessing a shared folder.

The easiest way to confirm the credential is to use Explorer to access remote server’s C$ share. Also, you can use “net use” command.

For those who want to connect to a remote server programmatically using WTSOpenServer API, WNetAddConnection2 function should be enough to make sure you got a piece of credential before calling WTSOpenServer function.

Of course the user in the credential should be a use on the remote server who has enough privilege to carry out the operation.

2. Open RPC ports on firewall
An API call involves connecting to a remote machine use RPC. In most cases the RPC service is running and you can confirm that from the service management interface. To open RPC ports, simply enable “File and Printer Sharing” in the Windows Firewall setting.

(Generally, if the RPC is blocked by the firewall on remote machine, the error should be "1722 RPC server is unavailable" rather then "5 Access denied")

3. Disable "Force Guest" log on

You can change this option from either local security policy or registery.

Local security policy : run secpol.msc > Secuirty Settings > Local Policies > Security Options > Network access : Sharing and security model for local accounts, and set to "Classic".

Registry : find "forceguest" item in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa, and set it to 0.

The two ways are equivalent. If this option is set to 1, remote logged in user will be impersonated as Guest account (and if Guest account is disabled, login fails). Set it to 0 enables user logging in as themselves.

4. If you still receive “access denied” and it only happens when the remote OS is Windows XP SP2, it is very probably the following case.

To confirm, you need a packet sniffer such as Wireshark (a great freeware).
When you use a sniffer to capture the SMB packets, you can see the authentication is OK (NTLM if workstation, Kerberos if Windows 2003 domain), but the RPC get nca_s_fault_access_denied (0×00000005) as error code.

This means the remote RPC component failed to execute the requested operation. It is because in Windows XP SP2, it is not allowed to operate on terminal service (Remote Desktop) through RPC by default. To modify this setting to enable Remote Desktop API through RPC, you need to find the following registry key:

HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server

Then add a DWORD value named “AllowRemoteRPC” and change its value to 1.

I googled it and it seems there is no official document on it. (I don’t know why)

Access denied error often misleads us into thinking it must be because of the insufficient privilege, or because that the qwinsta.exe command / WTSOpenServer failed to utilized the credential. It wasted me about 5 hours in researching how to attach a credential to a RPC call, and I even got caught deeply into the complicated SSPI/LSA authentication architecture.