Virtualization - Cloud

Month: March 2017

VM Backups failing on only one Node in a 2012R2 Cluster

Issue:

In a 5 node Hyper-v 2012R2 cluster,  all of sudden VM backups are failing on only one node(HOST2) i.e., backup team unable to take backup if any VM hosted on HOST2.

Observation:

  • When backup team is firing VM level backup on the HOST2, Backup is getting terminated with the VSS snapshot error..
  • If VM’s migrated to other node then backup is getting success for the same VM
  • Observed issue not specific to VM or any cluster shared volume -> Issue is occurring only if VM’s hosted on HOST2

Troubleshooting:

  • As issue specific to HOST2, tested VM backup with windows native backup tool -> Unable to take backup , terminating while creating VSS snapshot.
  • Created new VM on local D drive -> Tested with Windows backup tool -> Backup is getting success with windows backup tool if VM hosted on local drive, VM backup failing only if it is on Cluster shared storage
  • As issue specific to one server & CSV writer on HOST2 -> Started troubleshooting from the side of CSV writer
  • Done deep level analysis of event logs -> which indicates towards CSV writer unregistered -> Check below screenshot
  • Run the command “vssadmin list providers” on HOST2 and compared with other servers ->  it has been observed that  provider “Microsoft CSV Shadow Copy Provider”  is missing from HOST2 ->Screenshot attached
  • As CSV provider is missing on problematic HOST2 -> Fixed issue by exporting CLSID provider from working server and imported to HOST2 ->Check below screenshot
  • Post import , ran the “vssadmin list providers” -> Now provider list is same as working servers
  • Backup is working fine post fixing all..

Error Screenshots

 

 

Volume Shadow Copy Service (VSS) provides the ability to create a point in time image (shadow copy) that can be used to perform backups. In our environment, backup of VM failed immediately which was hosted on HOSt2 node, once it shows as “Snapshot Processing”. This means, snapshot operation is not happening. Provider ID(400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff) which is reflecting in Event viewer logs is related to MS CSV Shadow Copy Provider, which is not existing in registry editor as it might have unregistered.

Working Server(HOST3)

Not Working(HOST2) ->CLSID is missing

Final Screenshot

Ref:

 

One of the Hyper-V node in 2012R2 Cluster Changing to Pause Node Automatically

In one of my customer infra, we have 5 nodes in Hyper-v 2012R2 cluster. Among these 5 nodes , always Node1 changing to pause mode automatically for every 30 mins..

Issue:

Node1 is going to Pause State(With DO NOT FAIL ROLES BACK)  i.e., Node going to pause state without moving VM’s.

Observation:

  • Issue is getting resolve only after stopping of SCVMM agent service on BHHV-A01.
  • There was a recent migration happened(approx. 2 months) from Hyper-v 2012 to 2012 R2 and SCVMM 2012 to 2012R2
  • No schedule tasks were configured

In ideal scenario, Hyper-v will be go in pause mode only if administrator keep in maintenance mode or SCVMM will keep node in pause mode if it  is configured with  Dynamic optimization or PRO in SCVMM -> But, these settings are not configured in SCVMM

Issue looks very typically as only one node is having an impact and issue resolving if we stop SCVMM agent on Node1

I know that SCVMM is cluprit as issue resolving post stopping of SCVMM agent service -> I have asked customer to reinstall SCVMM agent on Node1 but he is not convinced.

Started searching SCVMM known issues in forums and found the below resolution

Solution:

It has been observed that, SCVMM was installed with RTM version in and there is a known  pause issue listed in Update Rollup 5.

Latest Rollup is Update Rollup 10 and below issue fixed in Rollup 5

Reference:

Pass-through disk addition in Highly Available VM – Difference in 2012 & 2008

Steps to add Pass-through Disk in Highly Available VM –  2012R2

  • Shutdown VM if it is powered on (Best Practice)
  • Make Sure Disk is online at HOST level and note down Disk Number
  • Go to Failover Console -> Add Disk to cluster ->After adding it will be placed in “Available Storage” ->Note the Disk number in console for later verification
  • Check whether Disk owner in failover console is displaying current working Server or not , else you need to proceed all steps by logging to the disk owner server.
  • In Failover Console ->Under Disk’s  section -> Right Click Disk ->  Assign to VM Role-> Select the VM which for which you want to assign.
  • After adding the disk to Failover Cluster, assign it to the VM role and ensure that the disk is online on HOST. If it is offline when you perform the remaining steps, the disk will be Read-Only in the VM with no way to fix it but to start over
  • In Failover ->Roles -> Go to VM -> Check under Resources Section -> Under Virtual Machine-> “Virtual Machine Configuration”  resource should online
  • In failover Console -> Go to VM Settings ->Add virtual SCIS adapter -> Pass Through Disk 4
  • Start VM ->Check Disk is accessible or not
  • Test Live Migration  

In 2008 or 2008 R2

DISK should be offline at HOST else it will go in READ-ONLY MODE  -> Blogs confirmed the same and I too seen the same issues

A new disk must be brought Online and Initialized before it can be used. This process writes a disk signature to the disk so cluster can use it. Once the disk has been initialized, it can be placed Offline again. No partitioning is required as that will be accomplished inside the virtual machine

Difference between adding pass-through disk in 2008 & 2012 is -> In 2008, Disk should be initialized and make offline whereas in 2012 it should be online throughout  the process

References

 

 

 

Pass-through Disk addition Issue in Cluster – Disk Read only issue after adding pass-through Disk

Issue

  • Unable to add Pass-through Disk in failover console to make Virtual Machine’s(2 VM’s) High Available with Pass-Through Disk.
  • Multiple VM’s had pass-through disk’s and no issue with any other.
  • Issue occurred  after one of my team member  removed pass-through disk post VM shutdown
  • Able to add pass-through disk without adding in to HA

Initial Troubleshooting

  • One of my team member removed pass-through disk and shutdown as part of planned maintenance activity – Post VM start , Disk went into Read-Only mode in guest O.S.
  • Due to less time,  without VM shutdown  I  tried to remove pass-through disk from VM ->Disk changing to Read-Only Mode
  • As disk changing to read-only mode, assumed that disk need to keep in offline at Host level ->Therefore, I had only option to change Disk to Turn-on Maintenance mode in failover console
  • In Failover console-> Kept Disk into Maintenance Mode -> Added pass-through disk to VM  in Failover Console -> Worked fine, Disk is in normal mode in Guest O.S.

Keeping the Disk into maintenance will not impact any functionality. Enabling of this mode will just disables few Disk checks like File/Device System Check’s , Is Alive, Look Alive etc. which performed by cluster service

Maintenance mode will remain on until one of the following occurs:

  • You turn it off.
  • The node on which the resource is running restarts or loses communication with other nodes (which causes failover of all resources on that node).

I have taken downtime as I need to Turnoff Disk Maintenance and resolve issue permanently

Next Troubleshooting:

  • Removed VM & Disk  from High Availability and Re-added to Failover Console -> No Luck
  • Moved VM to different Host server’s and tested the same steps to isolate issue from Host level ->No Luck
  • Created Test VM and executed similar to isolate issue from VM level ->No Luck
  • Tested by assigning Cluster disk’s with different servers to isolate issue from Disk ownership -> No luck
  • Tried Pass-through Disk by keeping in Disk Maintenance mode ( Previous state) -> No Luck
  • Removed VM & Disk from HA and added only in Hyper-v Manager -> It is working without High Availability 

Next Observations:

  • Before adding to cluster , when making DISK online -> Disk automatically coming with drive letter by appearing in windows Explorer -> Drive letter appearing as pass-through disk is not new(fresh), it is already using in production with drive letter so directly mounting.
  • When adding Cluster Disk in Failover console (let’s say  in HOST1) -> Disk ownership changing to HOST2 after adding to cluster -> This is the main difference which we made with current and  other VM’s  
  • Received Error while adding Pass -through Disk to VM  in Failover Console -> Error “An error occurred while updating the virtual machine configuration settings, Error code:0x8007100c, Not Supported”

Involved Microsoft support to check this typical issue, below are root cause & solution for the subjected issue

  • The UI(Failover Console) was trying to check the permission due to which we received an error on the disk which we are presenting as pass through as they are presented from SAN.
  • When we add the disk as pass through to the VM it gets added with the MPIO path of the disk. Due to which when we add it from the failover cluster manager to the VM it fails to update that path on the VM configuration file as it needs certain permissions, which it cannot see as we cannot add permission on the path \\?\mpio#disk&ven_dgc&prod_raid_5&rev_0532#1&7f6ac24}

Error:

‘Virtual Machine “DBL’ failed to start.

‘DBL’ failed to start. (Virtual machine ID XXXXXXXXXX)

‘DBL’ Synthetic SCSI Controller (Instance ID XXXXXXX): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID XXXXXXX)

‘DBL’: Hyper-V Virtual Machine Management service Account does not have permission to open attachment ‘\\?\mpio#disk&ven_dgc&prod_raid_5&rev_0532#1&}’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID XXXXXXX)

The highlighted is the path of the disk on which we cannot add the permission.

To force that path to get updated on the VM configuration file we have to run the PowerShell command.

update-clustervirtualmachineconfiguration -vmid XXXXXX-XXXX-XXXX

Above command updated the path succesfully in VM configuration, VM booted successfully

Error Screenshots

 

 

References:

How to add a Pass-through disk to a Highly Available Virtual Machine running on a Windows Server 2012 R2 Failover Cluster

Read-only pass-through disk after you add the disk to a highly available VM in a Windows Server 2008 R2 SP1 failover cluster

 

 

 

 

VM Registration Failure – VM Missing – VM Failed States in Failover Console

I have encountered below scenarios few months back  and issue addressed by following below approach

  1. VM appearing in Failover console and disappeared in Hyper-v Manager after unexpected reboot of Host server, it failed over and unregistered. -> Unable to find VM on any host in Hyper-v Manager
  2. After applying pass-through disk in failover console, it given access denied error but I started the VM to check -> VM tried to start in all nodes(16) and failed ->Post that, unable to start VM & open VM settings ->Unable to find VM on any Host in Hyper-v Manager

Troubleshooting

  • Restarted VMM service – No luck – Had a thought of reregistering with import option as event id said failed to unregister.
  • Imported With registration option From Hyper-v Manager – VM registered successfully

And for the 2nd case -> Registered  VM with Mklink command

  • Hyper-V operates using a list of symbolic links in a specific directory:  C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines
  • Each of these are links to the actual VM configuration files in their own respective subdirectories – whether stored locally or on shared storage, the link doesn’t change in its nature.
  • First you need to identify the GUID of the specific VM. 
  • As an example will use the LitwareSpeech VM, located at D:\VMs\LitwareSpeech.  In the “D:\VMs\LitwareSpeech\Virtual Machines” path is the configuration file for this VM, named “D546B942-76AF-4C3B-97C6-9EE74828BC91.XML”

Using the VM GUID that you determined above in Step 1, run the following command:

Syntax:   mklink <GUID>.XML <VMConfigPath.XML> or in our example

mklink D546B942-76AF-4C3B-97C6-9EE74828BC91.xml “D:\VMs\LitwareSpeech\Virtual Machines\D546B942-76AF-4C3B-97C6-9EE74828BC91.xml”

Above command restores the reference to your VM in Hyper-V Manager.

Ideally, when you create a VM Hyper-V creates a security entry (ACE) on this symbolic link for the SID of the worker process for the VM.  Unfortunately, this ACE isn’t re-created when you recreate the symbolic link using mklink as detailed above.

If you try to start your re-registered VM at this point, you may receive permission issue

To address this issue, follow these steps:

Using this GUID, run the following command to provide permissions

Syntax:  icacls “C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\<GUID>.xml” /grant “NT VIRTUAL MACHINE\<GUID>“:(F) /L

icacls “C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\D546B942-76AF-4C3B-97C6-9EE74828BC91.xml” /grant “NT VIRTUAL MACHINE\D546B942-76AF-4C3B-97C6-9EE74828BC91”:(F) /L

Above command regenerates the necessary ACE on the symbolic link using the Service SID of the VM, rather than on the configuration file itself, replicating the initial state of the symbolic link.

Once this command has been run successfully, you should be able to start your VM without further issues.

If above command steps does not work, you can try by stopping Hyper-v Virtual Machine Management Service

How to Unregister VM
  • In few scenarios, we may need to unregister
  • Delete the file symbolic link, will just deletes link and unregister from Hyper-v Manager. The VM and its configuration will be kept in your HDD
  • You can follow the above  steps to register VM

Error Screenshots

 

 

© 2024 Tech Blog

Theme by Anders NorenUp ↑