1. Setup test client machine with WinRM if not already installed. (WinRM is automatically installed with all currently-supported versions of the Windows operating system.)
2. Enable PowerShell remoting in the test client machine.
Do not worry about the warning that it will be removed. It will just be moved under release tab will be a more enhanced feature. The link specified in the warning message clearly explains it.
When adding machines you can use machine FQDN. Visual Studio Team Services need no direct access to these test client machines. Having one release agent setup in the domain, which have access to the test client machines, as shown in Diagram 01, would be sufficient. Release agent machine can access the test client machines with domain credentials since it is inside the on-premise domain.
4. Create a release definition to create test client, and set to use an agent in the on premise domain. Configure on-premise agent for Visual Studio Team Services explained here.
Specify the machine group name in Machines, created in step 3. Agent configuration user can be different from the user name provided for accessing the machines in machine group defined (same user used here). Agent configuration user is used to configure test agent, and test agent will run as this user. Check Interactive Process if Coded UI test needs to be run on the test clients.
Next post will explain how to run a coded UI test with the deployed agent.