Rafael Rivera
Twitter GitHub LinkedIn Email Feed

How to set up synthetic kernel debugging for Hyper-V virtual machines

Published ยท Last revised

Windows 8 (and Windows Server 2012) introduced a new debugging transport called KDNET. As the name implies, it allows kernel debugging over a network and can be faster and easier to set up than its predecessors (e.g. COM and IEEE 1394).

MSDN has great background information on setting up kernel debugging via Visual Studio and by hand, however, Microsoft's official stance on virtual machine debugging is to continue using the old and slow serial port. Even on generation 2 virtual machines. This makes some tasks, like dumping memory or resolving symbols, a slow and tedious task.

But unofficially you can instead use what's internally referred to as "synthetic debugging".

Figure 1 - Simplified Synthetic Debugging Architecture
Figure 1 - Simplified Synthetic Debugging Architecture

To understand how it works, first consider a common KDNET scenario on a physical machine. When KDNET is enabled, the Microsoft Kernel Debug Network Adapter built into Windows takes over (and shares) the physical network device installed in the machine. Communication to and from the machine (and kernel debugger) occurs over the network as expected and life is grand. But virtualized environments (in child partitions) add another layer of abstraction to the underlying hardware that presents a problem -- the Kernel Debug Network Adapter cannot latch directly onto the physical network device.

Cue the magic behind synthetic debugging.

To overcome the inability to directly control network hardware, KDNET was built with smarts to detect virtualized environments and switch to communicating over VMBus -- a inter-partition communication mechanism -- with the host operating system (in the parent partition), as shown in Figure 1. The parent then exposes a KDNET endpoint for you to communicate with the virtualized environment over the network. With this set up, the virtualized environment doesn't require network connectivity!

To set up synthetic debugging (for Windows), you need to be running:

  • Windows 8 or Windows Server 2012+ on the host side
  • Windows 8 or Windows Server 2012+ on the guest side (generation 1 or 2)

Here's the step-by-step:

  1. On the guest machine, open an elevated command prompt and issue the commands:

    bcdedit /dbgsettings NET HOSTIP:1.2.3.4 PORT:55555
    bcdedit /debug on

When KDNET kicks in and detects the virtualized environment,
the HOSTIP and PORT parameter values are ignored, in favor of the
VMBus.

  1. Copy the key value displayed and keep it handy.

  2. On the host machine, open an elevated Powershell instance and run the following script after adjusting the VMName and DesiredDebugPort values:

    $VMName = "Virtual Machine name here"
    $DesiredDebugPort = 55555

    $MgmtSvc = gwmi -Class "Msvm_VirtualSystemManagementService" `
        -Namespace "root\virtualization\v2"

    $VM = Get-VM -VMName $VMName
    $Data = gwmi -Namespace "root\virtualization\v2" `
        -class Msvm_VirtualSystemSettingData `
        | ? ConfigurationID -eq $VM.Id

    $Data.DebugPort = $DesiredDebugPort
    $Data.DebugPortEnabled = 1
    $MgmtSvc.ModifySystemSettings($Data.GetText(1))

Be sure to specify an open ephermeral (49152-65535) UDP port here.

When the machine is powered on (or reset), the virtual machine's specific Virtual Machine Worker Process (vmwp.exe) will attempt to bind to that port on all (management) interfaces.

  1. Power cycle the guest machine. A reset is not sufficient as it doesn't tear down the Virtual Machine Worker Process.

  2. On any machine on the network, connect a debugger to the Hyper-V host machine with the port and key from earlier. For example, to connect with WinDbg, issue the following command:

    windbg.exe -k net:target=hyperv-host,port=55555,key=a.b.c.d

When not configured correctly, synthetic debugging can be quite a pain to troubleshoot. Here are some tips if you run into problems:

  • Ensure the guest is running Windows 8 or above and the active BCD entry is configured correctly (via bcdedit).
  • Ensure the guest's Virtual Machine Worker Process has bound to the port you specified with TcpView or CurrPorts. This can be extremely problematic if your Hyper-V host is also a DNS server like mine.
  • Ensure no other virtual machines are configured to use the same port.
  • Ensure any firewalls in the way are configured to allow UDP traffic through via specified debug port.

Happy debugging!