Virtualization - Cloud

Day: March 18, 2017

Pass-through disk addition in Highly Available VM – Difference in 2012 & 2008

Steps to add Pass-through Disk in Highly Available VM –  2012R2

  • Shutdown VM if it is powered on (Best Practice)
  • Make Sure Disk is online at HOST level and note down Disk Number
  • Go to Failover Console -> Add Disk to cluster ->After adding it will be placed in “Available Storage” ->Note the Disk number in console for later verification
  • Check whether Disk owner in failover console is displaying current working Server or not , else you need to proceed all steps by logging to the disk owner server.
  • In Failover Console ->Under Disk’s  section -> Right Click Disk ->  Assign to VM Role-> Select the VM which for which you want to assign.
  • After adding the disk to Failover Cluster, assign it to the VM role and ensure that the disk is online on HOST. If it is offline when you perform the remaining steps, the disk will be Read-Only in the VM with no way to fix it but to start over
  • In Failover ->Roles -> Go to VM -> Check under Resources Section -> Under Virtual Machine-> “Virtual Machine Configuration”  resource should online
  • In failover Console -> Go to VM Settings ->Add virtual SCIS adapter -> Pass Through Disk 4
  • Start VM ->Check Disk is accessible or not
  • Test Live Migration  

In 2008 or 2008 R2

DISK should be offline at HOST else it will go in READ-ONLY MODE  -> Blogs confirmed the same and I too seen the same issues

A new disk must be brought Online and Initialized before it can be used. This process writes a disk signature to the disk so cluster can use it. Once the disk has been initialized, it can be placed Offline again. No partitioning is required as that will be accomplished inside the virtual machine

Difference between adding pass-through disk in 2008 & 2012 is -> In 2008, Disk should be initialized and make offline whereas in 2012 it should be online throughout  the process

References

 

 

 

Pass-through Disk addition Issue in Cluster – Disk Read only issue after adding pass-through Disk

Issue

  • Unable to add Pass-through Disk in failover console to make Virtual Machine’s(2 VM’s) High Available with Pass-Through Disk.
  • Multiple VM’s had pass-through disk’s and no issue with any other.
  • Issue occurred  after one of my team member  removed pass-through disk post VM shutdown
  • Able to add pass-through disk without adding in to HA

Initial Troubleshooting

  • One of my team member removed pass-through disk and shutdown as part of planned maintenance activity – Post VM start , Disk went into Read-Only mode in guest O.S.
  • Due to less time,  without VM shutdown  I  tried to remove pass-through disk from VM ->Disk changing to Read-Only Mode
  • As disk changing to read-only mode, assumed that disk need to keep in offline at Host level ->Therefore, I had only option to change Disk to Turn-on Maintenance mode in failover console
  • In Failover console-> Kept Disk into Maintenance Mode -> Added pass-through disk to VM  in Failover Console -> Worked fine, Disk is in normal mode in Guest O.S.

Keeping the Disk into maintenance will not impact any functionality. Enabling of this mode will just disables few Disk checks like File/Device System Check’s , Is Alive, Look Alive etc. which performed by cluster service

Maintenance mode will remain on until one of the following occurs:

  • You turn it off.
  • The node on which the resource is running restarts or loses communication with other nodes (which causes failover of all resources on that node).

I have taken downtime as I need to Turnoff Disk Maintenance and resolve issue permanently

Next Troubleshooting:

  • Removed VM & Disk  from High Availability and Re-added to Failover Console -> No Luck
  • Moved VM to different Host server’s and tested the same steps to isolate issue from Host level ->No Luck
  • Created Test VM and executed similar to isolate issue from VM level ->No Luck
  • Tested by assigning Cluster disk’s with different servers to isolate issue from Disk ownership -> No luck
  • Tried Pass-through Disk by keeping in Disk Maintenance mode ( Previous state) -> No Luck
  • Removed VM & Disk from HA and added only in Hyper-v Manager -> It is working without High Availability 

Next Observations:

  • Before adding to cluster , when making DISK online -> Disk automatically coming with drive letter by appearing in windows Explorer -> Drive letter appearing as pass-through disk is not new(fresh), it is already using in production with drive letter so directly mounting.
  • When adding Cluster Disk in Failover console (let’s say  in HOST1) -> Disk ownership changing to HOST2 after adding to cluster -> This is the main difference which we made with current and  other VM’s  
  • Received Error while adding Pass -through Disk to VM  in Failover Console -> Error “An error occurred while updating the virtual machine configuration settings, Error code:0x8007100c, Not Supported”

Involved Microsoft support to check this typical issue, below are root cause & solution for the subjected issue

  • The UI(Failover Console) was trying to check the permission due to which we received an error on the disk which we are presenting as pass through as they are presented from SAN.
  • When we add the disk as pass through to the VM it gets added with the MPIO path of the disk. Due to which when we add it from the failover cluster manager to the VM it fails to update that path on the VM configuration file as it needs certain permissions, which it cannot see as we cannot add permission on the path \\?\mpio#disk&ven_dgc&prod_raid_5&rev_0532#1&7f6ac24}

Error:

‘Virtual Machine “DBL’ failed to start.

‘DBL’ failed to start. (Virtual machine ID XXXXXXXXXX)

‘DBL’ Synthetic SCSI Controller (Instance ID XXXXXXX): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID XXXXXXX)

‘DBL’: Hyper-V Virtual Machine Management service Account does not have permission to open attachment ‘\\?\mpio#disk&ven_dgc&prod_raid_5&rev_0532#1&}’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID XXXXXXX)

The highlighted is the path of the disk on which we cannot add the permission.

To force that path to get updated on the VM configuration file we have to run the PowerShell command.

update-clustervirtualmachineconfiguration -vmid XXXXXX-XXXX-XXXX

Above command updated the path succesfully in VM configuration, VM booted successfully

Error Screenshots

 

 

References:

How to add a Pass-through disk to a Highly Available Virtual Machine running on a Windows Server 2012 R2 Failover Cluster

Read-only pass-through disk after you add the disk to a highly available VM in a Windows Server 2008 R2 SP1 failover cluster

 

 

 

 

VM Registration Failure – VM Missing – VM Failed States in Failover Console

I have encountered below scenarios few months back  and issue addressed by following below approach

  1. VM appearing in Failover console and disappeared in Hyper-v Manager after unexpected reboot of Host server, it failed over and unregistered. -> Unable to find VM on any host in Hyper-v Manager
  2. After applying pass-through disk in failover console, it given access denied error but I started the VM to check -> VM tried to start in all nodes(16) and failed ->Post that, unable to start VM & open VM settings ->Unable to find VM on any Host in Hyper-v Manager

Troubleshooting

  • Restarted VMM service – No luck – Had a thought of reregistering with import option as event id said failed to unregister.
  • Imported With registration option From Hyper-v Manager – VM registered successfully

And for the 2nd case -> Registered  VM with Mklink command

  • Hyper-V operates using a list of symbolic links in a specific directory:  C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines
  • Each of these are links to the actual VM configuration files in their own respective subdirectories – whether stored locally or on shared storage, the link doesn’t change in its nature.
  • First you need to identify the GUID of the specific VM. 
  • As an example will use the LitwareSpeech VM, located at D:\VMs\LitwareSpeech.  In the “D:\VMs\LitwareSpeech\Virtual Machines” path is the configuration file for this VM, named “D546B942-76AF-4C3B-97C6-9EE74828BC91.XML”

Using the VM GUID that you determined above in Step 1, run the following command:

Syntax:   mklink <GUID>.XML <VMConfigPath.XML> or in our example

mklink D546B942-76AF-4C3B-97C6-9EE74828BC91.xml “D:\VMs\LitwareSpeech\Virtual Machines\D546B942-76AF-4C3B-97C6-9EE74828BC91.xml”

Above command restores the reference to your VM in Hyper-V Manager.

Ideally, when you create a VM Hyper-V creates a security entry (ACE) on this symbolic link for the SID of the worker process for the VM.  Unfortunately, this ACE isn’t re-created when you recreate the symbolic link using mklink as detailed above.

If you try to start your re-registered VM at this point, you may receive permission issue

To address this issue, follow these steps:

Using this GUID, run the following command to provide permissions

Syntax:  icacls “C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\<GUID>.xml” /grant “NT VIRTUAL MACHINE\<GUID>“:(F) /L

icacls “C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\D546B942-76AF-4C3B-97C6-9EE74828BC91.xml” /grant “NT VIRTUAL MACHINE\D546B942-76AF-4C3B-97C6-9EE74828BC91”:(F) /L

Above command regenerates the necessary ACE on the symbolic link using the Service SID of the VM, rather than on the configuration file itself, replicating the initial state of the symbolic link.

Once this command has been run successfully, you should be able to start your VM without further issues.

If above command steps does not work, you can try by stopping Hyper-v Virtual Machine Management Service

How to Unregister VM
  • In few scenarios, we may need to unregister
  • Delete the file symbolic link, will just deletes link and unregister from Hyper-v Manager. The VM and its configuration will be kept in your HDD
  • You can follow the above  steps to register VM

Error Screenshots

 

 

© 2024 Tech Blog

Theme by Anders NorenUp ↑