Hyper-V integration services, are a bundled set of software which, when installed in the virtual machine improves integration between the host server and the virtual machine. Integration services (often called integration components), are services that allow the virtual machine to communicate with the Hyper-V host. Hyper-V Integration Services is a suite of utilities in Microsoft Hyper-V, designed to enhance the performance of a virtual machine’s guest operating system.
In short and general, the integration services are a set of drivers so that the virtual machine can make use of the synthetic devices provisioned to the VM by Hyper-V.
Hyper-V Integration Services optimizes the drivers of the virtual environments to provide end users with the best possible user experience. The suite improves virtual machine management by replacing generic operating system driver files for the mouse, keyboard, video, network and SCSI controller components. It also synchronizes time between the guests and host operating systems and can provide file interoperability and a heartbeat.
Below is the list of Integration Services Version numbers
Windows Server 2008
|Build Number||Knowledge Base Article ID||Comment|
|6.0.6001.17101||n/a||Windows Server 2008 RTM|
|6.0.6001.18016||KB950050||Windows Server 2008 RTM + KB950050|
|6.0.6001.22258||KB956710||Windows Server 2008 RTM + KB956710|
|6.0.6001.22352||KB959962||Windows Server 2008 RTM + KB959962|
|6.0.6002.18005||KB948465||Windows Server 2008 Service Pack 2|
|6.0.6002.22233||KB975925||Windows Server 2008 RTM + KB975925|
Windows Server 2008 R2
|Build Number||Knowledge Base Article ID||Comment|
|6.1.7600.16385||n/a||Windows Server 2008 R2 RTM|
|6.1.7600.20542||KB975354||Windows Server 2008 R2 RTM + KB975354|
|6.1.7600.20683||KB981836||Windows Server 2008 R2 RTM + KB981836|
|6.1.7600.20778||KB2223005||Windows Server 2008 R2 RTM + KB2223005|
|6.1.7601.16562||n/a||Windows Server 2008 R2 Service Pack 1 Beta|
|6.1.7601.17105||n/a||Windows Server 2008 R2 Service Pack 1 RC|
|6.1.7601.17514||KB976932||Windows Server 2008 R2 Service Pack 1 RTM|
Windows Server 2012
|Build Number||Knowledge Base Article ID||Comment|
|6.2.9200.16384||n/a||Windows Server 2012 RTM|
|6.2.9200.16433||KB2770917||Windows Server 2012 RTM + KB2770917|
|6.2.9200.20655||KB2823956||Windows Server 2012 RTM + KB2823956|
|6.2.9200.21885||KB3161609||June 2016 update rollup for Windows Server 2012|
Windows Server 2012 R2
|Build Number||Knowledge Base Article ID||Comment|
|6.3.9600.16384||n/a||Windows Server 2012 R2 RTM|
|6.3.9600.17415||Windows Server 2012 R2 RTM + KB3000850|
|6.3.9600.17831||KB3063283||Windows Server 2012 R2 RTM + KB3063283|
|6.3.9600.18080||KB3063109||Windows Server 2012 R2 RTM + KB3063109|
|6.3.9600.18339||KB3161606||June 2016 update rollup for Windows Server 2012 R2|
|6.3.9600.18398||KB3172614||July 2016 update rollup for Windows Server 2012 R2|
|6.3.9600.18692||June 27, 2017—KB4022720 (Preview of Monthly Rollup)|
The files used by Hyper-V VM are as below: In short, to explain:
The BIN file created in the virtual machine folder of the virtual machine is equal to the size of the memory of the virtual machine and is a placeholder to save the virtual machine state in the event that the Hyper-V host shut down.
The BIN file contains the memory of a VM and is located inside the GUID folder. If the VM in powered off state, there will be no BIN file present. This file is the equal to the size of the VM’s memory provisioning in Hyper-V Manger.
In Windows Server 2008 and Windows Server 2008 R2 – starting a virtual machine would result in Hyper-V creating a .BIN file which matched the size of the memory assigned to the virtual machine. Microsoft did this to ensure that we always had enough disk space available to create a saved state (which is particularly critical if the physical computer is shutting down – and the virtual machine is configure to save state when the physical computer shuts down).
The BIN file is simply idle while the virtual machine is powered on; it is pre-allocated so that its space is guaranteed to be available if needed and for quicker response to a save action. However – many people did not like to see their disk space being “wasted” like this.., as BIN file is idle during running state.
To address this, since Windows “2012” Microsoft made a simple change: Hyper-V only pre-create the .BIN file if you choose “Save the virtual machine state” as the Automatic Stop Action for the virtual machine. If you choose “Turn off the virtual machine or Shut down the guest operating system”, BIN file will not create with equal size of RAM.
It is still possible to save the state manually as long as there is enough room for the file. Above Automatic Stop Action setting in only applicable when Physical computer shutdown.
By default, all virtual machines have an Automatic Stop Action of Save, which means the state of the virtual machine saved to disk. However, the best practice is once Integration Services are enabled the Automatic Stop Action should be changed to “Shut down the guest operating system”, which performs a clean shutdown and no longer needs the BIN file to save the memory content to.
Steps to save storage space by removing BIN File
Local Host Cache (LHC) & Evolution
Local Host Cache was a core feature of the Independent Management Architecture (IMA) that was introduced with Citrix Metaframe XP 1.0 in 2001, and was still used until Citrix XenApp 6.5 and now reintroduced in XenApp/Desktop 7.12
Technically, the LHC is a simple Access database where it stores a subset of the data store in each Presentation (XenApp) server. The IMA service running on each Presentation(XenApp) Server downloads the information for every 30 mins or whenever a configuration change is made in farm.
LHC primary functions are permits a server to function in the absence of a connection to the data store & improves performance by caching information of applications.
LHC contain the information of servers, published applications, Domain & Licensing. LHC evolved a lot over the years and allowed SQL downtimes for an indefinite period in its last release with XenApp 6.5.
If the data store is unreachable, the LHC contains enough information about the farm to allow normal operations for an indefinite period, if necessary. However, no new static information can be published, or added to the farm, until the farm data store is reachable and operational again.
The disappearance of LHC
With the release of the awful version 7.0 of XenApp in 2013 and the move to XenDesktop FlexCast Management Architecture (FMA), Citrix decided to remove the Local Host Cache feature–and many others–without offering any other alternative. To be fair, Citrix converged XenApp into XenDesktop, which was already using the FMA design since the version 5 and without Local Cache Host equivalent. This decision immediately made the SQL infrastructure a critical piece of any XenApp implementation. Any downtime on the SQL infrastructure would immediately cause a downtime for new sessions on the XenApp infrastructure as well. It could also have some side effects with the old Citrix Web Interface.
Citrix recommends having a highly available SQL infrastructure to host XenApp and XenDesktop databases. While you can successfully implement HA for your SQL infrastructure, it does not necessarily mean that you will avoid downtimes, as many components are to be considered.
Facing a storm of complaints, Citrix also started–finally!–to listen to its customers and released XenDesktop 7.6 in Sept 2014 with the Connection Leasing (CL) feature enabled by default.
Unfortunately, CL was not full replacement of LHC and it is alternative option provided in placement of LHC, limited to frequently used and assigned applications/desktops (up to 2 weeks by default). For users not using Citrix frequently or using pooled desktops, CL is completely useless and did not resolve anything. There are also many limitations: load management, workspace control, power actions are not supported.
Citrix came up with a milestone achievement with its new idea as part of the XenDesktop 7.12 release in Dec 2016. This time, they claimed to bring back all the Local Host Cache (LHC) features from XenApp 6.5, even adding few improvements to make it more reliable. LHC feature is offered for Cloud and On Premises implementations along Connection Leasing in 7.12, but is considered the primary mechanism to allow connection-brokering operations when database connectivity to the site database is disrupted. Surprisingly, Local Host Cache feature is disabled by default. Let us expect Citrix to enable that feature by default in the next version.
When installing XenDesktop 7.12 and up, a SQL Express instance(Local DB) will be installed locally on each Delivery Controller to store the Local Host Cache. Config Synchronizer Service (CSS) takes care of the synchronization between the remote database and the Local Host Cache (Local DB). The Secondary Brokering Service (Citrix High Availability Service) takes over from the Principal Broker when an outage is detected and does all registration and brokering operations.
There are many limitations to consider with this version of LHC
In conclusion, it took Citrix almost 4 years to deliver a somewhat equivalent of the good old Local Host Cache for XenDesktop 7.x. The database is not a single point of failure anymore in a XenDesktop/XenApp deployment. However, customers with large deployments are not supported with this version of the Local Host Cache and some of the -HUGE- limitations can discourage you from using that feature
PVS stream service abrupt termination intermittently (approx. once in month) which causing user sessions freeze and user unable to launch HSD’s.
Explained to customer that missing of best practices will lead to these type of intermittent issues , since there is no root cause found and it is not a best practice to keep cache on server in production environment , prepared a plan to change cache configuration to” Cache on RAM overflow to HDD”.
Current PVS Storage configuration for cache as below
PVS1 (VM)->1700 GB allocated through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)
PVS2 (VM) -> 1700 GB allocated through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)
Proposed Storage change Configuration as below:
Post referring multiple blogs, Write Cache proposed to all images(profiles) is 20 GB -> Therefore , for 86 HSD, 1820 GB required and it should present to complete Hyper-v cluster as HSD hosted on cluster.
How to change Cache Mode for existing production Vdisk:
To replicate to other devices(VM’s)
Since exact root cause was not found, vendor provided & self-analyzed would be the cause for abrupt streaming service termination
When each Target Device boots up the OS is not aware of the Write Cache and writes to the logical disk that it is presented (the vDisk). The PVS driver then redirects this data at the block level to the write cache which in your environment is held on the PVS server. When BNIStack driver ( transport stack for communicating with the PVS server) is then initialized, it will pull down chunks of the vDisk as and when they are needed from your vDisk store which is on your PVS server also. The BNIStack driver will also access the write cache directly when it needs to issue additional writes from the target device. The communication in relation to the above is carried out between the BNIStack driver on the target device and the Stream Service on the PVS server. The case notes state that environment have approximately 2000 users which I suspect is putting a large overhead on the Stream Service responsible for servicing all Write Cache requests.
The Target Device BNIStack driver is also responsible for retries (because UDP does not). The base timeout for a packet timeout is 10 seconds. If the server responds quickly, this value is reduced by half all the way down to 1 second. Correspondingly, if the server responds slowly the timeout will double all the way up to 10 seconds.
A retry timeout of 1 second or less may cause excessive I/O retries, leading to slow response and hanging of target devices which will then ultimately lead to a stream service failure. Since we know we are seeing issues with the stream service and subsequent hanging/crashing of this service and considering we are aware of the large write cache overheads incurred on the stream service already, I suspect that the load we are putting on the stream service is too great, culminating in it grinding to a halt, this will then present itself with the symptoms we see at the target device e.g. slow logging in, sluggishness inside the session.
As explained above, the recommendation to mitigate this is to move the Write Cache away from the PVS Server itself and place the processing overhead at the target device instead.
Recommendations & Reference
Citrix provides servicing options to give greater flexibility and choice in how to adopt new XenApp, XenDesktop, and XenServer functionality while giving greater predictability for maintaining and managing the support of your environment
Last year, Citrix introduced two new XenApp / XenDesktop servicing options, the LTSR, which stands for Long Term Service Release and the CR a.k.a. Current Release., In 2016, Citrix announced first LTSR of XenApp and XenDesktop 7.6 and in 2017 first LTSR for XenServer 7.1 that is available for download on Citrix.com.
What is LTSR?
As a benefit of Software Maintenance, Long Term Service Releases (LTSR) of XenApp ,XenDesktop,XenServer enable enterprises to retain a particular release for an extended period of time while receiving minor updates that provide fixes, typically void of new functionality. Long Term Service Releases (LTSR) is ideal for large enterprise production environments where you would prefer to retain the same base version for an extended period
A Long Term Service Release guarantees 5 years of mainstream support and an optional 5 years of extended support (needs to purchased separately). This includes cumulative updates every 4 to 6 months, a new LTSR version of XenApp / XenDesktop every 12 to 24 months and any potential (hot) fixes
A valid Software Maintenance (SM) contract is needed to make use of the LTSR or CR servicing option.
Ideal customer environment for a LTSR is for the customers who typically follow a 3-5 year version upgrade cycle
Long Term Service Releases will have a regular cadence of Cumulative Updates that will typically contain only fixes
What is Current Release?
Any new release of XenApp/XenDesktop/XenServer will be labeled a Current Release. With the CR servicing option you can always make use of (install) the most recent XenApp and/or XenDesktop versions including all the latest enhancements and additions that come with it.
Its release cycles are much shorter with a new version release being announced every three to nine months in general.
Citrix recommends that large enterprise customers have a combination of Current Release and Long Term Service Release environments.
Switching from a LTSR to a CR servicing, and vice versa, is always optional as well
All initial releases of XenApp/XenDesktop/XenServer will be a Current Release. There will likely be multiple Current Releases of a major XenApp/XenDesktop/XenServer version (i.e. 7.6, 7.6 FP1, 7.6 FP2, 7.6 FP3, 7.7, 7.8 ,7.9,7.11,7.13,7.14); however, there will likely only be one LTSR release of that version after that release is considered customer-tested and industry-proven (i.e. 7.6 FP3).
How will the customer know if their environment is Long Term Service Release compliant?
Citrix support and engineering have developed the LTSR Assistant tool which will scan your environment and compare your environment with the necessary LTSR components to determine if you are compliant. The tool provides a report that will outline the necessary updates to achieve compliance. The LTSR Assistant tool is available for download athttp://support.citrix.com/article/CTX209577.
Will a customer running an LTSR compliant environment be supported if they also have non- compliant components?
Citrix does not recommend mixing non-compliant components. For example, if a customer decides to implement Provisioning Services 7.7, which is not compliant with the current 7.6 LTSR environment and the customer has an issue with Provisioning Services 7.7 the customer may be asked to move to the latest Provisioning Services Current Release to receive public fixes
How often will Citrix release a Long Term Service Release of XenApp and XenDesktop or XenServer?
Citrix will release a Long Term Service Release of XenApp and XenDesktop or XenServer based on the number of features, implementations, customer support cases and general feedback. However, as very general guidance it can be expected that Citrix will release a new Long Term Service Release every 12-24 months; however, Citrix reserves the full rights to alter those timelines.
Is Citrix discontinuing the process of providing Hotfix Rollup Packs (HRP) for XenApp and XenDesktop?
With LTSR, Cumulative Updates will replace Hotfix Rollup Packs (HRP). Hotfix Rollup Packs (HRP) will still be made available for XenApp 6.5.
Will 7.6 LTSR support XenApp for Windows Server 2008 R2 for 10 years?
Windows Server 2008 R2 will not be eligible for extended support. Citrix will continue to monitor Windows 2008 R2 lifecycle dates for future determination of lifecycle milestones.
In one of our customer infra, for one of the VM ,snapshot grown to 1.9 TB size and it was created by one of the engineer as part of IS upgradation but forgot to delete.
Due to above 2 challenges, planned below options and completed as part of prerequisites
Note: If Merge process taking more than expected, we cannot can cancel in between the merge process as there are lot of chance corruption
Roll Back Plan:
|Before Execution||Post Snapshot Deletion
VM File Size
|VHD File||Drives letters in OS||Parent File||Snapshot File||Total VM||Storage Volume|
|Time taken for Deletion of Snapshot 1.9 TB in Offline is 5 hrs. (12:30 to 5:30 A.M), space reclaimed is 1885.8 GB|
In a 5 node Hyper-v 2012R2 cluster, all of sudden VM backups are failing on only one node(HOST2) i.e., backup team unable to take backup if any VM hosted on HOST2.
Volume Shadow Copy Service (VSS) provides the ability to create a point in time image (shadow copy) that can be used to perform backups. In our environment, backup of VM failed immediately which was hosted on HOSt2 node, once it shows as “Snapshot Processing”. This means, snapshot operation is not happening. Provider ID(400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff) which is reflecting in Event viewer logs is related to MS CSV Shadow Copy Provider, which is not existing in registry editor as it might have unregistered.
Not Working(HOST2) ->CLSID is missing
In one of my customer infra, we have 5 nodes in Hyper-v 2012R2 cluster. Among these 5 nodes , always Node1 changing to pause mode automatically for every 30 mins..
Node1 is going to Pause State(With DO NOT FAIL ROLES BACK) i.e., Node going to pause state without moving VM’s.
In ideal scenario, Hyper-v will be go in pause mode only if administrator keep in maintenance mode or SCVMM will keep node in pause mode if it is configured with Dynamic optimization or PRO in SCVMM -> But, these settings are not configured in SCVMM
Issue looks very typically as only one node is having an impact and issue resolving if we stop SCVMM agent on Node1
I know that SCVMM is cluprit as issue resolving post stopping of SCVMM agent service -> I have asked customer to reinstall SCVMM agent on Node1 but he is not convinced.
Started searching SCVMM known issues in forums and found the below resolution
It has been observed that, SCVMM was installed with RTM version in and there is a known pause issue listed in Update Rollup 5.
Latest Rollup is Update Rollup 10 and below issue fixed in Rollup 5
Steps to add Pass-through Disk in Highly Available VM – 2012R2
In 2008 or 2008 R2
DISK should be offline at HOST else it will go in READ-ONLY MODE -> Blogs confirmed the same and I too seen the same issues
A new disk must be brought Online and Initialized before it can be used. This process writes a disk signature to the disk so cluster can use it. Once the disk has been initialized, it can be placed Offline again. No partitioning is required as that will be accomplished inside the virtual machine
Difference between adding pass-through disk in 2008 & 2012 is -> In 2008, Disk should be initialized and make offline whereas in 2012 it should be online throughout the process