Booting ARM64 builds of Windows 10 in QEMU

QEMU boot Windows 10... on ARM(64)

As you may know, the venerable Quick Emulator (QEMU) supports emulation of the AArch64/ARM64 architecture. With some fiddling over the weekend, I was able to boot and install arm64 builds of Windows 10.

QEMU actually running Windows 10... on ARM(64)

Here's how I did it (feedback welcome):

  1. Warning: This is slow as dirt on an Intel Core i7 4770K

  2. Warning: I have not yet compiled the VirtIO drivers for network and other ancillary devices. That means these machines don't support networking.

  3. Download and install QEMU for Windows

  4. Download the Windows 10 (arm64) ESDs from adguard's whizzbang download page and glue them together using UUPtoISO (patched for arm64) to create a usable ISO

  5. Download my hand-crafted UEFI firmware and recompiled/signed arm64 storage drivers

  6. Create a system.vhdx that's around 23GB or larger (fixed size, not expanding, initialized using GPT partitioning scheme)

  7. Generally glue all the above together in a folder somewhere and create a windows.cmd with the following contents:

  8. Run the script.

  9. During setup, you will need to provide VirtIO drivers (browse to the mounted disk).

  10. Complete setup as usual. (This will take a long time.)

For those curious, here's the break down of the QEMU arguments, in order of appearance:

I encourage you to read the QEMU documentation for additional options.

Tips:

  • Add --accel tcg,thread=multi for additional per-core performance gains (Thanks @never_released!)
  • The Windows guest may panic at device detection. Click skip when given the option and you should slowly get through.
  • The Windows guest may intermittently panic with a Kernel Security Check Failure.
  • At first boot, the Windows guest's Compatibility Telemetry runner process will spin up and eat all your CPU cores. Kill the process and performance should greatly increase.

Adding Acrylic Blur to your Windows 10 (Redstone 4) desktop apps

Back in July 2015, I provided a code sample for turning on what, at the time, I called "Aero Glass" for Windows 10 desktop apps. (Since then, its name was revealed to be Acrylic Blur.) This code was circulated to the far corners of the Internet and even made it into a shipping product at a very large company.

That's pretty cool.

But the underlying undocumented API has now changed and the sample code I provided early no longer functions correctly on late Redstone 4 builds.

Let's talk about what changed.

First, the AccentState enumeration changed, gaining a new constant specifically designed for Acrylic Blur use.

Second, the Accent Policy GradientColor member is now expected to be non-zero. Despite its unfortunate name (pulled straight from Windows symbols), this member controls the blur background and must be set to an RGBA value stored in little-endian word order (ABGR).

With some minor tweaks to the original sample, Acrylic Blur is back. You can find an updated sample app on GitHub.

acrylicblur_sample-1

Reminder: This API is not supported and can change at any time.


(02/02/2018: Thanks @StartIsBack for hitting me over the head and clarifying a revision of this post that initially discussed a two-word GradientColor.)

What's that gobbledygook in my Package Family Name?

Appx applications have a notion of a "package family" in which all the individual packages belong to. For example, the Windows Photos app may ship on x86, x64, ARM32, ARM64, and IA64 platforms and each package belongs to the overall "Microsoft.Windows.Photos_8wekyb3d8bbwe" family.

A package family name consists of two parts, separated by an underscore: A Name (e.g. Microsoft.Windows.Photos) and a Publisher ID (e.g. 8wekyb3d8bbwe).

The Name is typically derived from a common Package Name (specified in the Appx Manifest) or specified manually by a developer.

The Publisher ID is a Crockford Base32-encoded representation of the first 8 bytes of the SHA-256 hash of the Publisher UTF-32 encoded string. Phew.

For example:

  • Publisher: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
  • SHA-256 hash: 471D3F2C6D42D7C7FACEEF1ADD20881817082D593D8B7B3E229B3617E4CD2A18
  • ... first 8 bytes: 47 1D 3F 2C 6D 42 D7 C7
  • Crockford Base32 encoded (no padding bytes): 8wekyb3d8bbwe

Using the Base3264-UrlEncoder package available on Nuget (Eric Lau, Mhano Harkness), you can write a few lines of code to calculate this yourself.

Windows Container App Compat: Access host COM ports from a Windows Server container

I field many questions from folks trying to containerize messy legacy apps. Some of the more interesting apps have been line-of-business apps with arcane licensing requirements (e.g. physical dongle) and radio communication apps (e.g. those requiring a GPS).

Windows Server does not ship in a configuration that allows for Windows Server containers to access host COM ports.

But you can change that.

Unlike Hyper-V containers, which rely on a special virtual machine for kernel-level isolation, Windows Server containers simply use Windows Server Silos to isolate processes from the host machine. Windows Server ships with a default silo configuration that aligns it with container mantras around container ephemerality, immutability, scability, etc. You can tweak that configuration to meet your needs.

For example, here's a step-by-step to bring in a COM port:

  1. Take ownership of \Windows\System32\Containers\wsc.def and give yourself write access. (Make a backup copy!)

  2. Open up wsc.def with your favorite text editor.

  3. COM port symlinks typically live in the Global?? object directory, so add the appropriate symlink under <objdir name="GLOBAL??" ...>. (For example, using WinObj, I determined my COM1 is a symlink to \Device\Serial0 and added <symlink name="COM1" path="\Device\Serial0" scope="Global" />.)

  4. Add the appropriate symlink for the device under <objdir name="Device" ...>. (Continuing the example above, I added <symlink name="VCP0" path="\Device\VCP0" scope="Global" />)

  5. Save the file and relaunch your container. (The vmcompute process reads the rules at container startup.)

  6. Type mode at the command prompt to verify your COM port is now visible.

  7. Brace yourself for disappointment when updates come down from WU in the far future and blow away your changes.

As you've probably figured out by now, you can expose way more than just COM ports -- almost anything can be exposed to containers. Be careful.

I did some limited testing on my end with a FTDI serial cable connected to a radio. Shoot me an email if this worked (or failed) for you.

KRACK: Impacted devices I use

More of a TODO list than a blog post, but wanted to document what I use, that is potentially affected by KRACK, and its associated patch status. In other words, I'm doomed.

Last updated: 10/31/2017 10:12AM PST

  • ✔️ Xbox One - Assumed Fixed.

  • ❌ Nintendo Switch - Awaiting vendor response

  • ❌ Nintendo 3DS - Awaiting vendor response

  • ❌ Sony PlayStation 3 and 4 - Awaiting vendor response

  • ❌ Amazon Echo - Awaiting vendor response

  • ❌ Amazon Kindle - Awaiting vendor response

  • ❌ Samsung Galaxy S8 - Awaiting vendor/mobile operator (T-Mobile) response. Struck out with technical support; emailed Samsung PR at 10/16 2:20PM PST. Also ref# 1135145703.

  • ✔️ iPhone 8 - Fixed

  • ✔️ Windows PCs - Fixed.

  • 〰️ Surface devices - Windows 10 is patched, however it's not immediately clear if wireless chipset drivers (for Modern Standby use) are safe.

  • ✔️ macOS PCs - Fixed

  • ❌ Samsung K-series TV - Awaiting vendor response. Struck out with technical support; emailed PR at 10/16 2:20PM PST.

  • ❌ Brother Printer - Awaiting vendor response. Stuck out with technical support; left voicemail for legal department at 10/16 1:50PM PST.

  • ❌ Netgear Nighthawk R7000 - Confirmed affected. Awaiting firmware.

  • ❌ Synology NAS - [Confirmed affected] (https://www.synology.com/en-us/support/security/Synology_SA_17_60_KRACK). Awaiting update.

Windows Container App Compat: opencv-python on Windows Server Core

When attempting to install and load the opencv-python package, you may run into the following error on Windows Server Core images (e.g. python:2.7-windowsservercore):

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python\lib\site-packages\cv2\__init__.py", line 9, in <module>
    from .cv2 import *
ImportError: DLL load failed: The specified module could not be found.

This error pops up because opencv relies on ancient Video for Windows, Microsoft AVI, and Microsoft ACM Audio components, none of which ship with the Windows Server Core base image.

If you need to use parts of opencv that don't rely on these components, copy the following libraries from your container host into the container's \System32 folder (assuming 64-bit usage here, adjust accordingly).

  • msacm32.dll
  • avifil32.dll
  • avicap32.dll
  • msvfw32.dll

Docker Recipe: Media Foundation on Windows Server Core

Applications, like Plex Media Server, rely on the Media Foundation Platform for access to codecs, hardware devices, and DirectX Video Acceleration. Media Foundation is built into consumer versions of Windows, but is an optional bolt on for Windows Server for obvious attack surface/footprint reasons.

Installing MF in optional environments like Server Core is typically easy enough. A simple PS> Install-WindowsFeature, a reboot, and you're done. Container authors, however, will struggle with Media Foundation for several reasons:

  1. Media Foundation packaging is marked with 'reboot required' metadata
  2. Component Based Servicing (CBS) respects this metadata and goes down the path of configuring the system for pending service operations (e.g. \WinSxS\pending.xml) to run at power on/off
  3. Of course, servicing the base image isn't a supported scenario, so the components involved (poqexec.exe) fail (e.g. NtFsControlFile failures in Windows::Rtl::TxfLogExpander::GetCurrentContainerCountMax) and CBS never gets to finish

The Media Foundation package authors clearly had a reason to mark this package as "reboot required" but that may have been before the advent of containers and the minimal subsystem within.

If you want to experiment with the Media Foundation Platform package in a container (unsupported), you will need to modify the package metadata to not require a reboot prior to installing the package.

Here's a contrived PowerShell example you can execute in your Dockerfile:

Get-ChildItem "$Env:SystemRoot\Servicing\Packages\*Media*.mum" |
ForEach-Object {
  (Get-Content $_) -replace 'required','no' | Set-Content $_
}

If you prefer to be explicit (recommended), you'll want to patch up:

  • Microsoft-Windows-Media-Format-Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum
  • Microsoft-Windows-Server-Media-Foundation~31bf3856ad364e35~amd64~~10.0.14393.0.mum

Feel free to reach out if you run into any other snags. You can find me in the Windows Containers forum on MSDN or contact me using the links on the blog.

Office apps and the Mso::ChecksumRegistry::Data algorithm

Nearly all Microsoft apps have experimentation code built-in these days. This code is typically used to facilitate A/B testing externally but internally, it can problematic if left on. (Imagine trying to reproduce a bug in a feature that's turning itself on/off or behaving differently across machines.) This is why most apps also house a feature or experiment control panel, hidden to all except those typically on the owning team.

Story Remix has a panel, for example, that appears when a specifically named .json file is present on disk and populated with specific values. Groove Music relies instead on specialty registry values locked down to specific geographic markets. Office apps also use the registry, but with an added obfuscation twist.

I haven't fully mapped out how Office apps store all their experimental settings, but we're going to talk about one area today:

LocalState\HKCU\Software\Microsoft\Office\16.0\Common\ExperimentTas\[app]\Flights

The registry value names are straightforward -- they represent an arbitrary name assigned to an experiment or feature and hardcoded into the app. The registry value data represents a structure, serialized to a UTF-16 string, in the form [field]|[fieldType]|[fieldValue];[settingType]|[settingValue]. (This is then stuffed into a composite value, hence the 0x5f5e10c type and extra bits at the end that are out of scope for the blog post.)

A more concrete example:

  • Value: Microsoft.Office.Experimentation.ABC
  • Data: Mso::ChecksumRegistry::Data|uint64_t|9215221962101605702;bool|1

The signed (yes, despite the declaration) 64-bit checksum value is computed by taking the value name, setting type, and setting value, concatenating them together, and running it through the Fowler–Noll–Vo (variant 1a) hashing function.

Using our example data above, let's walk through this:

  1. Glue the pieces together: Microsoft.Office.Experimentation.ABCbool|1
  2. Run it through once with hardcoded initial offset basis: step2 = FNV1a(string_bytes, 0xCBF29CE484222325)
  3. Use that as the initial offset basis for another run, with a hardcoded chunk of data this time:
    hash = FNV1a([0xb7, 0x3c, 0x75, 0x96, 0xd5, 0x79, 0x13, 0xa5, 0xc8, 0xe5, 0xb2, 0xa2, 0xaf, 0x2b, 0x44, 0xff], step2)
  4. Serialize the hash and stuff it into the pattern above

If you implement this correctly, with say the Microsoft.Office.Experimentation.ShowExperimentSettings registry value, you'll end up with a hash of 2507011508917168823.

And a shiny new control panel to play with.

hxmail_experiments_panel

Registry values in the LocalState\HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\ExperimentEcs\[app]\Flights key use a different type of obfuscation that I'll cover in the next post.

Notes on swapping the internal Xbox One hard disk for a solid-state disk

In a desperate effort to improve the Xbox One user experience (UX), I experimented with swapping out the Samsung Spinpoint M8 5400 RPM hard disk for a solid-state disk (SSD). My hypothesis at the time was that the disk, because of its slow speed, must be very busy and that swapping it out would yield a noticeable performance boost to the operating system and improve overall UX.

It didn't.

But a few folks on Twitter expressed interest in reproducing the experiment so here are my notes. (Thanks for your patience, Stefán!)

Hardware Considerations

  • All Xbox consoles ship in storage configurations (e.g. 500GB, 1TB) that have this configuration burnt into its flash memory. This configuration is used to rebuild the partition layout on the internal disk in recovery scenarios, presenting an obstacle for a quick and easy disk swap.

    That means an Xbox One 500GB will always want to restore a partition layout compatible with a 500GB disk.

  • All Xbox consoles ship with a SATA2 controller. The Xbox One S ships with a SATA3 drive but the same controller, limiting the drive to theoretical SATA2 speeds. This part swap was likely due to the scarcity and current cost of SATA2 disks.

  • The wireless chipset reports antenna status to the operating system. If this is not plugged in, you will not be able to complete the out-of-box-experience (OOBE), even in a wired configuration.

Software Considerations

  • Encrypted container use is common on this platform. This presents a hypothetical hardware configuration data persistence problem when migrating data from one disk to another.

  • The boot loader appears to maintain state about previous successful/unsuccessful boots, which could lead to unexpected behavior when swapping disks. (More testing is needed in this area.)

  • Anti-rollback protection is present and used, preventing use of older versions of the OS after an update. This can invalidate hard disk backups very quickly.

  • The disks are set up with a standard GUID Partition Table layout, with strict validation of both header and partition array CRC32 checksums.

  • All partitions must also be assigned a well-known GUID. This should not be confused with the partition type GUID.

  • I did not test if the disk identifier was also used/validated, but it's not unreasonable to assume such.

  • I did not test if the backup GPT header (backup LBA) was used/validated.

  • The GUID Partition Table entry order is not validated. You can re-order partitions, given you continue to meet the requirements above.

    As the Temp and User partitions are likely to be most busy, it makes sense to stuff those on the larger, outer hard disk tracks. Their adjacency also allows for short disk head travel for the inevitable back and forth.

    But programmatically generating the User partition on non-standard disks can be tricky due to being positioned in the middle of the table. Some opted for simplicity and round to the nearest gibibyte and ignore what's left. PowerShell made this easy to implement accurately but it's not necessary.

    It's possible I missed the one application that retrieves an enumerable list of partitions and selects a partition using a hard-coded index. But the odds of that kind of code surviving a code review at Microsoft are high.

  • Software updates to the Xbox One OS may not be compatible with the new storage, I have yet to receive an update to my test device. I did, however, enable dev mode successfully with no side effects.

Tools Used

Procedure

  1. Disassemble the Xbox One and remove the disk.
  2. Use a USB to SATA bridge to copy the contents of the disk to the PC.
  3. Use partitioning script to ready a blank SSD for Xbox One use.
  4. Copy the contents of the disk (i.e. each partition) to the newly prepared SSD
  5. Plug the SSD into the Xbox One and boot.
  • The Xbox One may exhibit odd behavior at this point. It may boot but report free space incorrectly, or it may not boot at all. This is the hypothetical hardware configuration data persistence problem I was referring to. To fix this, I reset the console.
  1. Restore the Xbox One to its factory defaults. Be sure to complete OOBE and gracefully shut down the Xbox One after that's completed.
  • At this point, Xbox One has restored the original partition layout on the disk, which is not what you want. (See considerations above.) But the characteristics of our disk should be implanted within a container somewhere, which is great.
  1. Remove the SSD from the Xbox One.
  2. Use a USB to SATA bridge to copy the contents of the SSD to the PC.
  3. Use partitioning script to wipe and ready the SSD for Xbox One use again.
  4. Copy the contents of the SSD back to the prepared blank SSD.
  5. Plug the SSD into the Xbox One and boot.
  • The Xbox One should now report the available free space correctly and everything should be functioning normally. Be sure to read the considerations above for potential future gotchas.

Adding the "Aero Glass" blur to your Windows 10 apps

(Update: At the time of writing, Fluent had not yet been fully revealed. We now know this to be the start of Acrylic Blur work.)

Since the reintroduction of Aero Glass in Windows 10, I've been receiving questions on how to incorporate that functionality into 3rd party applications. A few nights ago I looked into it and here's my guidance:

  1. Abandon code that uses DwmEnableBlurBehindWindow. This function hasn't been deprecated, oddly enough, but is effectively dead on Windows 10. Stop using it. (Also consider abandoning DwmExtendFrameIntoClientArea.)

  2. Start using SetWindowCompositionAttribute directly. It's not officially documented but here's the plumbing you need, if you're writing C# utilizing Interop Services tooling:

  3. On the view side, you don't need to worry about chroma keys anymore! Simply ensure your window uses a (background) brush with an alpha channel and the compositor will handle the rest.

The sample project I used to create the screenshot above can be found on GitHub.

Have fun!