Shared VMDK Disk Between Multiple VMs on VMWare

2.4K

VMware allows you to share the same virtual disk (.VMDK file) between 2 (or more) virtual machines running on different VMWare ESXi hosts or between VMs on VMware Workstation. Two or more virtual machines can read and write to a shared VMDK file simultaneously. This configuration is most commonly used in clustering scenarios.

VMware provides several options for the organization of shared disk resources for clustering scenarios (such as Oracle RAC and Windows Server Failover Cluster/WSFC):

• Multi-writer VMDK — this disk operation mode allows you to share the same VMDK file between multiple VMs (cluster nodes).
• Shared VMDK on clustered datastore — starting with vSphere 7.0, it is recommended to use VMDKs on clustered datastore as shared disk resources instead of multi-writers for WSFC clusters.

VMware Multi-Writer Mode for Shared VMDK

By default, VMware does not allow multiple virtual machines to access the same .vmdk file that is located on the shared datastore (VMFS, NFS, vSAN, or VVol). Virtual machine file locks prevent access to other virtual machines’ hard disks and avoids data corruption caused by multiple writers on the non-cluster-aware filesystems.

Hint. Multi-writer is also used for VMWare Fault Tolerance when primary and standby virtual machines access a shared vmdk file simultaneously.

The main restrictions of shared VMWare disks in Multi-Writer mode:

• You can’t online migrate running VMs with shared disk to another host (vMotion), or datastore (Storage vMotion). You can migrate between ESXi hosts only powered off virtual machines.
  

  Hint. vMotion is only supported for Oracle RAC cluster with up to 8 ESXi hosts;

• If you try to write data to such a vmdk from the guest operating system (for example, creating a folder), then this data (folder) will be visible only on the host that created it. Those, the Multi-Writer VMDK technology is needed just for clustering, and it’s impossible to use it as a shared disk with automatic file synchronization;
• You cannot expand such a VMDK disk online (if you try to increase the size of the vmdk, you get the error: The disk extend operation failed: The virtual disk requires a feature not supported by this program;
• You can’t suspend a VM with a shared disk;
• Snapshot creation, Changed Block Tracking, and vSphere Flash Read Cache (vFRC) are not supported;
• Virtual disks in multi-writer mode must not be connected to the virtual NVMe controller;
• Do not hot add a virtual hard disk with the multi-writer flag enabled, as this will cause it to be removed;
• In vSphere 6.7+, you can simultaneously share a single VMDK in multi-writer mode with up to 64 hosts. This behavior is supported for VMFS5/VMFS6 datastores as well after you enabled the /VMFS3/GBLAllowMW config option

Sharing a VMDK Disk Between Two Virtual Machines on VMware ESXi Hosts

Suppose you have two ESXi hosts with a shared datastore. These hosts are running two VMs (node1 and node2) that you want to add a shared virtual disk to.

First, you need to add a new SCSI controller on both virtual machines.

In the vSphere Client inventory select the first virtual machine (Node 1), and select Actions > Edit Settings.

Select Add New Device > SCSI controller in the dropdown list, and press the Add button.

Select LSI Logic SAS as the type of SCSI controller. Be sure to create a new SCSI controller, don’t use a default SCSI 0 controller.

Note. Check how to configure autostart of VM on VMware ESXi.

Now you need to choose the SCSI Bus sharing mode:

• Virtual — if you want to share a virtual disk between VMs on the same ESXi host;
• Physical — used when you need to share VMDK file between VMs on different ESXi hosts.

Hint. None mode is used only for non-clustering Multi-writer drives.

Press OK. Next, you need to add a new virtual disk on the first VM (Add New Device > Hard Disk > Add) with the following disk settings:

• VM Storage Policy: optional;
• Location: you need to select a shared datastore name that will store the vmdk file;
• Disk Provisioning: select Thick provision eager zeroed (shared virtual disk cannot be zeroed thick or thin provisioned. You can use other disk provisioned types if your shared vmdk file is located on the vSAN 6.7 Patch 01 datastore or vVOLs datastore).
  

  Note. You can convert thick to thin in VMWare ESXi;

• Sharing: Multi-writer;
• Virtual Device Node: select the SCSI controller created earlier;
• Disk mode: Independent – Persistent – in this mode, you can’t create snapshots for a VMDK you want to share.

Hint. If you enabled VMWare Fault Tolerance protection for a virtual machine, the multi-write VMDK mode is enabled automatically for virtual disks.

In the same way, you need to add a new SCSI controller to another virtual machine (Node 2). Then you need to add an existing new hard disk in the settings of the second virtual machine.

Select the VMFS Datastore, which stores the shared vmdk file that you created earlier for the first VM. Select the name of the first virtual machine, and in the middle pane select the desired vmdk file.

You will also need to enable the Multi-Writer sharing mode for this virtual disk.

Note. Single SCSI controller can connect up to 16 hard drives. This means that you don’t have to create an additional SCSI virtual controller every time you want to add a new shared VMDK.

Make sure you use the same SCSI device address for a shared vmdk disk on both virtual machines. For example, if you have a new address SCSI(1:0) on VM (node1) for a new disk (this means the first disk on the second SCSI controller), you should use the same address SCSI(1:0) for the shared disk on the second VM (node2). It is important!

Note. In ESXi versions, before ESXi vSphere 6.0 update 1, the MultiWriter mode for vmdk is also supported, but you cannot enable it from the vSphere Web Client interface. You can add a multi-writer flag by shutting down the VM and manually editing the vmx file of the virtual machine, adding the following line at the end of the file:

SCSI1:0.sharing = “multi-writer”

You can also add this parameter in the VM properties: Options > General > Configuration Parameters: Scsi1:0:sharing “multi-writer”.

After changing the virtual machine’s settings, connect to the console of the guest OS virtual machine. In this case, the VMs running Windows Server, so you can remotely connect to them using RDP (how to enable RDP remotely). Run the Computer Management console (compmgmt.msc), and expand Storage > Disk Management section. Right-click and select Rescan Disk.

The system detects the new disk and offers to initialize it. Select the desired partition table (GPT or MBR), create a new partition, and format it. Similar operations must be performed on the second VM. After these settings, both VMs are using a shared disk. Now you can proceed to set up a cluster solution based on Oracle RAC and Microsoft Cluster Services (MSCS).

Note. Keep in mind that using multi-writer VMDKs for non-clustered scenarios is not supported. If the shared VMDK disk is online on both ESXi hosts, you won’t see changes made to the disk by one virtual machine in the other virtual machine. This will happen until you bring the disk offline and then bring it online.

When trying to migrate a VM with a connected vmdk disk in MultiWriter mode, an error will appear:

Virtual machine is configured to use a device that prevents the operation: Device ‘SCSI controller X’ is an SCSI controller engaged in bus-sharing.

To resolve this issue turn off the VM, and perform cold vMotion.

In a similar way, you can provide simultaneous access from different VMs to the same raw LUN via Raw Device Mapping (RDM) mode. In this case, the RDM disk is being added to node1 as usual, and when adding a disk to node2 instead of selecting the LUN number, you need to select the mapping VMDK file for the RDM disk on the VMFS Datastore.

Using Shared Clustered VMDK For Windows Server Failover Clustering

Support for a clustered VMDK in a datastore is available in vCenter 7.0 or later. This mode should be used instead of the multi-writer feature for clustered disk resources in Windows Server Failover Clustering (including Failover Clysters and Microsoft SQL Server Always-On Failover Cluster Instances).

Shared clustered VMDKs can be used if:

• ESXi 7.0+ is installed on all hosts connected to the datastore;
• Storage must support ATS, SCSI-3 PR–type Write Exclusive–All Registrant;
• The datastore is formatted using VMFS 6.

You can enable the Clustered VMDK feature in the datastore settings. Go to Storage view > Configure > General > Datastore capabilities > Clustered VMDK > Enable.

Then add a new SCSI controller to your VM. Select LSI Logic SAS or VMware Paravirtual, and set SCSI Bus Sharing to Physical.

Then add a new VMDK disk with following settings:

• Disk provisioning: Thick Provision Eager Zeroed
• Virtual Device Node: select your new SCSI controller

Then open the second VM settings and add the new SCSI controllers and the select existing hard disk from the first VM. Select Virtual Device Node = New SCSI controller.

You’re now ready to set up a Windows Server Failover cluster on these VMs.

Note. The maximum number of WSFC nodes with Shared Clustered VMDK is 5.

Configuring Shared Disk in VMware Workstation

You can also share a VMDK disk between two running virtual machines on VMware Workstation. This mode only works in the Pro version and is available starting with VMware Workstation 12.

You can create a shared disk in VMware Workstation from the command prompt using the vmware-vdiskmanager.exe utility. Open cmd.exe as an administrator and run the commands:

mkdir SharedVMDK

cd SharedVMDK

Create a new 5 GB VMDK disk:

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -c -s 5GB -a lsilogic -t 4 shareddisk01.vmdk

Creating disk ‘shareddisk01.vmdk’

Create: 100% done.

Virtual disk creation successful.

Now you need to add the vmdk file (disk) you created to the first virtual machine. Open the settings of the virtual machine: Add > Hard Disk > SCSI disk type > Use an existing virtual disk > select file shareddisk01.vmdk.

In the Converting existing virtual disk to new format window, select Keep Existing Format.

Save your VM settings.

Now select the Hard Disk you added and click the Advanced button. Select the mode Independent > Persistent for the SCSI controller.

Save the VM settings disk. Do not turn on the virtual machine immediately. You need to manually edit the virtual machine configuration file (VMX). Run the command:

Notepad "C:\VHD\ClusterVM1\ClusterVM1.vmx"

Add the following lines to the VMX configuration file:

scsi0.sharedBus = "virtual"

disk.locking = "false"

Save the file.

Now add the shared VMDK file to the second virtual machine in the same way. Edit the VMX file of the second VM and add the same lines.

Start both VMs. When they run, a warning message will appear:

Clustering is not supported for VMware Workstation. This setting will be ignored.

Ignore this message. Now you can set up Windows Server Failover Cluster (WSFC) or Oracle RAC in your virtual machines running on VMware Workstation. The shared VMDK disk will be available in both VMs.

In this post, we discussed how to share a VMDK disk between multiple VMware VMs using Multi-Writer mode. This configuration is used for cluster-aware applications that know how to work with such shared disks.