VMotion CPU Compatibility - Migrations Prevented Due to CPU Mismatch - How to Override Masks

Vmotion migration preventing due to CPU Mismatch

Details

This article provides overriding VMotion CPU compatibility restrictions between ESX Servers.

Solution

If you have recent Intel 45nm Core 2 CPUs and were directed to this article by a message in VirtualCenter, refer to VMotion errors between different Intel 45nm Core 2 revisions (1008315) before continuing.

By default, VirtualCenter only allows migrations with VMotion between compatible source and destination CPUs. This article provides information about when it is appropriate to override CPU-compatibility constraints, how to override them, and provides links to additional information and resources. Topics include:

Background

Compatible CPUs for VMotion purposes means that source and target CPUs must have the same manufacturer (Intel or AMD) and be members of the same basic processor family, Pentium 4 or Core, for example. Within a given processor family, the source and target CPUs must also have certain common and extended features implemented (or not implemented, depending on the specific feature) for VMotion to succeed.

To determine if the source and target CPUs meet VMotion requirements, VirtualCenter compares the target CPU to a default bit mask definition. The default bit mask has evolved with each version of VirtualCenter to support (or preclude) VMotion migration given a specific set of CPU features. For example, the VirtualCenter 1.x bit mask does not flag the NX (Intel) or XD (AMD) bits, but the VirtualCenter 2.x bit mask does.

At the same time, manufacturers (Intel and AMD, specifically) continually improve their CPUs, releasing new features that may or may not be compatible with VMotion.

To circumvent VMotion’s CPU-compatibility checking for a specific feature or extended feature, you can modify the default bit mask. The bit mask-modification process (sometimes referred to in various VMware documents as “relaxing” a particular constraint) varies according to VirtualCenter Server version and ESX version.

A full discussion of the underlying components and how VirtualCenter Server and ESX Server host systems handle them is beyond the scope of this article. For more information, see CPU Compatibility Masks and Configuring Virtual Machines in the Basic System Administration Guide. Another useful resource is the technical paper on VMware VMotion and CPU Compatibility.

For specific information about Intel and AMD CPUs and VMotion compatibility, see:

For simplicity sake, modifications to the default bit mask for CPU compatibility for VMotion are also referred to simply as masks, typically identified by the CPU feature—for example, the SSE3 mask in this and related KB articles 1991 and 1992.

Scope of Modifications

The process of modifying VMotion compatibility settings depends on the VirtualCenter version and the ESX Server host system running the virtual machines. Also varying by VirtualCenter Server version and ESX Server version, is the scope of modifications to the default bit mask, as follows:

  • VirtualCenter 1.x—Modifications to the default bit mask apply to all ESX Server 2.x hosts being managed by VirtualCenter.
  • VirtualCenter 2.x with ESX Server 3.x—Modifications must be made on a per-virtual-machine basis using the VI Client.
  • VirtualCenter 2.0.1 Patch 2 (and later) with ESX Server 2.x or ESX Server 3.x—Modifications are made at the VirtualCenter Server level but can be defined by processor manufacturer, by version number, and other criteria.
  • VirtualCenter 2.5.x (and later)—Modifications to the CPU mask must be made while the virtual machine is powered off due to a risk of application or guest operating system instability.
  • VirtualCenter 2.5.0 Update 2 (and later)—Adds support for Enhanced VMotion Compatibility(EVC). EVC masks are applied before per-virtual-machine masks and global VirtualCenter server masks.

CPU Identification Tools

To obtain information about a host system’s CPU, you can use the appropriate vendor’s tool or VMware’s CPUID tool, depending on the system:

This KB discusses modifying the default bit mask in several different environments.
Warning : VMware does not support or recommend modifying the VMotion constraints for CPU extended features because of the risk of application or guest operating system failure after migration.


VirtualCenter Server 1.x

VirtualCenter Server 1.x bit mask applies to all virtual machines on the ESX Server 2.x host system. VirtualCenter Server 1.x uses a locally stored text file, config.ini , for configuration information.

Modifying the Default Bit Mask

To modify the default bit mask for all ESX Server 2.x hosts managed by VirtualCenter Server 1.x, make the changes to the VirtualCenter Server’s configuration file (config.ini ). The process of editing the config.ini file is generally the same (regardless of CPU vendor):
  1. Log in to the VirtualCenter Server as an administrator.
  2. Navigate to the VirtualCenter configuration directory. By default this directory is C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter
  3. Open the config.ini file in a text editor.

    Note: If the config.ini file does not exist, create it.
  4. Add the appropriate line for the CPU feature that you want masked to the config.ini file. See the Table below for more details.
  5. Save the file and restart the VirtualCenter Management Server Service so that the modified bit mask takes effect. The next time VMotion is attempted, the specific feature is ignored by VirtualCenter, and VMotion proceeds.
CPU Feature VirtualCenter Server 1.x Vendor Supported?
SSE3 migrate.ignore.extfeature.bits = 0xE5BD Intel, AMD No
SSSE3 migrate.ignore.extfeature.bits = 0x4E7BC Intel No

migrate.ignore.feature.bits = 0x20000

SSE4.1 migrate.ignore.extfeature.bits = 0x8E5BC Intel No
Single-core-Dual-core migrate.ignore.feature.bits = 0x90000000 AMD Yes
PERF_MSR migrate.ignore.extfeature.bits = 0xE5A0 Intel Yes


As an example, to effectively enable VMotion between a host based on an Intel CPU that has SSE3 and one that does not,add:

migrate.ignore.extfeature.bits = 0xE5BD

Warning: VMware does not support or recommend modifying the VMotion constraints for CPU extended features because of the risk of application or guest operating system failure after migration.

VirtualCenter Server 2.x and 2.5.x

VirtualCenter Server 2.x provides two different ways to modify VMotion CPU constraints, depending on the version of ESX Server being used:

  • For ESX Server 3.x hosts CPU constraints can be modified:

    • On a per-virtual-machine basis through the VMware Infrastructure Client (VI Client) graphical user-interface.

      Note: As of VirtualCenter 2.5.x, modifications to the CPU mask of an individual virtual machine must be made while the virtual machine is powered off. The ability to edit the CPU mask while powered on has been disabled because of the risk of application or guest operating system instability.

    • globally through the VirtualCenter Configuration file (vpxd.cfg ), as long as the host is being managed using VirtualCenter 2.0.1 Patch 2 and above.

  • For ESX Server 2.x hosts CPU constraints can only be modified globally by editing the the VirtualCenter configuration file (vpxd.cfg ).

    Note: To use global CPU masks VirtualCenter must be version 2.0.1 Patch 2 or later.

Per-Virtual-Machine Masking with ESX Server 3.x

In Virtual Center 2.x, the default bit mask for VMotion constraints among ESX Server 3.x systems can be applied on a per-virtual-machine basis using the VI Client. Regardless of CPU type or mask modifications, accessing the necessary dialog boxes is generally the same, as follows:

  1. Launch VI Client and connect to the VirtualCenter Server as an administrator.
  2. Click the Virtual Machine that you want to migrate from Inventory.
  3. Click Edit Settings on the Summary tab. The Properties window for the virtual machine displays.
  4. Click the Options tab in the Properties window.
  5. Click the CPUID Mask option to display the CPU Identification Mask information and settings.
  6. Click Advanced to display several settings-related boxes, including CPU Identification Mask, in the right-hand pane.
From here the mask can be modified appropriately for the selected virtual machine, depending on what needs to be masked. The following sections include steps on modifying the masks for several common CPU registers:

Modifying Default NX/XD Mask

Issues with the NX/XD features on CPUs are common. Ensure that the feature is either enabled (or disabled) in the BIOS of all physical ESX Servers to avoid these compatibility error messages.

Note: There may be no option to enable or disable the mask on all the physical servers.
If a CPU feature compatibility issue with the NX/XD bit is encountered, an error similar to the following is generated:

The CPU of the host is incompatible with the cpu feature requirements of virtual machine; problem detected at CPUID level 0x80000001 register 'edx'.
To modify the mask to enable or disable the NX/XD CPU bit:
  1. Navigate to the virtual machine Options tab (see steps above, if necessary).
  2. Select Hide the NX flag from guest or the Expose the NX flag to guest to disable or enable this CPU compatibility check for the selected virtual machine.
  3. Click OK to save the change.

    Note: The NX/XD mask can only be modified when the virtual machine is powered off.

Modifying Default Mask

Several other masks can be modified, to provide further compatibility for VMotion. To modify other CPU masks:
  1. Navigate to the virtual machine Options tab (see steps above, if necessary).
  2. Click Advanced to open the CPU Identification Mask properties dialog box.

    Note: The CPU Identification Mask dialog box has two tabs—Virtual Machine Default, and AMD Override. Most modifications for Intel CPU features are made on the Virtual Machine Default page. Modifications for AMD CPU features are made on the AMD Overrides page and are discussed in the next section of this article.

  3. Click the Virtual Machine Default tab to activate the dialog box, if necessary.

To modify the mask for a specific feature, enter the series of dashes and 0s as shown in the table below.

Feature Level Row Mask
SSE3 1 ecx

---- ---- ---- ---- ---- ---- ---0 -0-0

SSSE3 1 ecx

---- ---- ---- -0-- ---- --0- ---0 -0--

1 edx

---- ---- ---- --0- ---- ---- ---- ----

SSE4.1 1 ecx

---- ---- ---- 0--- ---- ---- ---- ----

When all modifications are complete, click OK and exit the CPU Identification Mask dialog box.

Modifying AMD-Specific Masks

Features that are specific to AMD processors are listed and can be changed on the AMD Override tab. To modify other CPU masks for AMD Processors:
  1. Navigate to the virtual machine Options tab (see steps above, if necessary).
  2. Click Advanced to open the CPU Identification Mask properties dialog box.
  3. Click the AMD Override tab to activate the dialog box. The AMD Override page displays.

To modify the mask for a specific feature, enter the series of dashes and 0s as shown in the table below.

Feature Level Row Mask
SSE3 1 ecx
---- ---- ---- ---- ---- ---- ---- ---0
RDTSCP 80000001 edx
---- 0--- ---- ---- ---- ---- ---- ----
80000001 ecx
---- ---- ---- ---- ---- ---- ---- 0---
CMPXCH16B
1 ecx

---- ---- ---- ---- --0- ---- ---- ----

FFXSR 80000001 edx
---- --0- ---- ---- ---- ---- ---- ----
3DNPREFETCH 80000001 ecx
---- ---- ---- ---- ---- ---0 ---- ----

When all modifications are complete click OK and exit the CPU Identification Mask dialog box.

Combining Mask Modifications

You can combine modifications to the default bit mask to allow migration with VMotion between groups that are incompatible based on more than one CPU feature. For example, to filter-out the compatibility check for both SSE3 and SSSE3 combine the following:

---- ---- ---- ---- ---- ---- ---0 -0-0

and:

---- ---- ---- -0-- ---- --0- ---0 -0--

to yield:

---- ---- ---- -0-- ---- --0- ---0 -0-0

for the ecx register mask.

Global Masking with ESX Server 2.x and ESX Server 3.x

For ESX Server hosts managed by VirtualCenter 2.0.1 Patch 2 (and subsequent releases), the bit mask can be modified globally by manually editing the VirtualCenter configuration file (vpxd.cfg ). The VirtualCenter 2.x-series configuration file, vpxd.cfg , contains XML tags defining various elements and settings for the VirtualCenter server. Bit-masks can be modified by adding the appropriate XML tags to define the mask in the context of a guest operating system configuration option to the configuration file .
Note: The vpxd.cfg file is located by default in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter
Although a full discussion of building these elements is beyond the scope of this article, the About the Tags section provides a brief overview. See the following section for specific steps related to Editing the VirtualCenter Configuration File and XML tags for Common Mask Patterns like, SSE3 , SSSE3 , and SSE3 and SSSE3 Combined .

About the Tags

The tags identify the register to which the mask applies, and for which hosts and which versions the mask applies. The tags are direct descendants of the tag, in the vpxd.cfg file. Tags are listed in the order in which they must be nested:
  • —Direct descendant (child) tag of element. The vpxd.cfg can contain a single guestOSDescriptor element.
  • <host-product-and-version> —Between the opening and closing tags, you can nest multiple host-product-and-version-tags that specify the version and host to which the masks you define apply. For ESX Server hosts, the tags can be specified generally, as in for an ESX Server 2.x host, for an ESX Server 3.x host or can be specified precisely— , for example. The tag identifies various maintenance release levels of the ESX Server to which the subsequently nested mask applies.
  • —Allows for the configuration to be restricted by virtual machine hardware version. For example, , , or . In the example below is specified, meaning that the mask applies to all virtual machine hardware versions.
  • —Allows for a guest version to be specified. For example, , , , and so on. In the example shown below, is specified, and therefore the mask applies to all guest versions.
  • —This tag precedes the actual mask. The mask is then defined for , , or , depending on your needs. Tag elements that are used to define the mask, identify the CPU mask. The element details include vendor, CPU ID level, and the CPU register. The valid choices for these are:
CPU vendor
, ,
CPU ID level
, , ,
CPU register
, , ,

Tags must be embedded in the order shown in the table. A CPU-vendor tag is followed by a CPU-ID-level tag, followed by a CPU-register tag. Immediately following the CPU-register tag is the sequence of 32 dashes and x-s that represent the actual 32-bit register mask appropriate for the feature being modified. The mask is then followed by the CPU-register-, CPU-ID-level-, and CPU-vendor-closing tags.

Note: A mask defined using tag is used by the system only when no vendor-specific mask has been specified for the same CPU ID level.
  • When you have finished defining masks, close with .
Below is an example of the tag order, shown in the context of the vpxd.cfg file. The and opening and closing tags are already in the vpxd.cfg file. The opening tag can be placed directly after the opening tag, embed the appropriate tags to suit your needs, followed by the closing tag.







[Elements and mask definition go in here. Common Mask Patterns can be copy-and-pasted from the below example patterns.]





...
In this example, any mask embedded in the [Elements and mask...] placeholder section applies to all ESX Server 2.x hosts being managed by the VirtualCenter Server, and to all guest operating systems.
Below are some sample XML snippets that can be copied and pasted into a vpxd.cfg file for the common mask modification patterns . VMware does not support or recommend modifying the VMotion constraints for CPU extended features because of the risk of application or guest operating system failure after migration.

Editing the VirtualCenter Configuration File

To edit the VirtualCenter Configuration File:
  1. Navigate to the location of the vpxd.cfg file. By default, the vpxd.cfg file is located in the C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter directory.
  2. Use a text editor (such as WordPad) to open the vpxd.cfg file. The vpxd.cfg file is an XML-formatted text file that contains various optional settings for the VirtualCenter Server.
  3. Add opening and closing tags to identify the hosts, versions, and other specifics to which the mask applies.
  4. Copy and paste from the examples in Common Mask Patterns, or apply other masks as needed for your deployment.
  5. When the appropriate changes are made, restart the VMware VirtualCenter Server Service to enable the new masking.

Common Mask Patterns

The following are example masks for several common CPU feature masks. Copy, paste, and modify them as required for the environment. VMware does not support or recommend modifying the VMotion constraints for CPU extended features because of the risk of application or guest operating system failure after migration.


SSE3



----:----:----:----:----:----:---x:-x-x




----:----:----:----:----:----:----:---x





SSSE3



----:----:----:-x--:----:--x-:---x:-x--
----:----:----:--x-:----:----:----:----







----:----:----:-x--:----:--x-:---x:-x-x
----:----:----:--x-:----:----:----:----




----:----:----:----:----:----:----:---x
Labels: ,