Microsoft is embracing and extending Wi-Fi Display

As part of a recent open specifications update, Microsoft has revealed it's extending the Wi-Fi Display specification to improve a number of wireless display scenarios. These extensions are described in the MS-WFDPE-Preview and MS-WDHCE-Preview specifications and include additions like a low-latency stream to carry mouse cursor data, a method of managing desired latency, and better error reporting.

Let's take a look at each of the additions.

Dynamic resolution and refresh rate

In scenarios where the source device changes video stream resolution or refresh rate -- think gaming -- devices normally require Real-Time Streaming Protocol (RTSP) renegotiation or, more often, freak the hell out and require you to restart your streaming experience. To smooth that over, Microsoft is introducing a method of detecting these changes (and a way for devices to report they support such). More specifically, devices that report support for this feature will monitor the H.264 stream's sequence parameter set/picture parameter set (SPS/PPS) for changes in resolution and frame rate and will adapt seamlessly.

Latency management

When it comes to latency, we typically think lower is better. But that's not always the case. For example, gamers require low latency to minimize input lag. Because video frames are pumped through as fast as possible, it's common for some tearing or artifacts to appear. But movie viewers don't care about latency. They want a pixel perfect jitter-free viewing experience. To achieve that, devices may extend their input buffer and hold onto video frames longer, a method that introduces a measurable but completely acceptable amount of latency.

The device manufacturer's dilemma surfaces here: Do they optimize for gaming? Or for casual movie viewers? Or do they release two SKUs of the same hardware with slightly tweaked software?

To overcome this huge pain point, Microsoft is introducing a capability for devices to receive a "latency mode" from source devices. The idea is that the source will have the context and responsibility of communicating the user's intended use of the wireless display. For example, the source could detect which app is in use (e.g. Windows Media Player or Microsoft PowerPoint) and send the appropriate latency mode (e.g. high or low, respectively).

Separate mouse stream

Wi-Fi Display is pretty simple in terms of its inputs. It supports one stream that is chock full of audio and video data. That works great for movies but not so much for scenarios involving input. And that's especially true for a mouse.

Microsoft is introducing a capability that will eliminate the move-the-mouse-and-wait game by decoupling the mouse from the video completely. This works by enabling a source device to send a separate mouse stream to a target device. The receiving side would then be responsible for combining the mouse cursor data with whatever is being displayed on the screen at the time.

If this sounds familiar, that's because Microsoft already does something very similar as part of its Remote Desktop Protocol (RDP).

Error reporting

You're streaming a game then poof, the stream is dead. What happened? From the perspective of the source device, you lost connection to the target. It knows something happened. But that's all the information you're going to get. Good luck troubleshooting that.

Microsoft is introducing a more formal method of reporting error details back to the source device. Supported devices will be on guard for "teardown" requests and provide reasons to enhance diagnostics and improve overall usability.

Richer metadata

Microsoft is also opening up some metadata enhancements made by Intel for its Intel Wireless Display (WiDi) solution. (These enhancements are listed in the tightly controlled Intel WiDi Specification.) Devices can use Intel-defined fields to report back rich metadata such as a friendly name, support URL, version, and logo.

Device support

From an operating system perspective, most of these features are available for use in Windows 10 Technical Preview. But my testing indicates no devices currently implement the new capabilities. This is likely a sign of a Microsoft Wireless Display Adapter update on the horizon.