Local Host Cache (LHC) & Evolution
Local Host Cache was a core feature of the Independent Management Architecture (IMA) that was introduced with Citrix Metaframe XP 1.0 in 2001, and was still used until Citrix XenApp 6.5 and now reintroduced in XenApp/Desktop 7.12
Technically, the LHC is a simple Access database where it stores a subset of the data store in each Presentation (XenApp) server. The IMA service running on each Presentation(XenApp) Server downloads the information for every 30 mins or whenever a configuration change is made in farm.
LHC primary functions are permits a server to function in the absence of a connection to the data store & improves performance by caching information of applications.
LHC contain the information of servers, published applications, Domain & Licensing. LHC evolved a lot over the years and allowed SQL downtimes for an indefinite period in its last release with XenApp 6.5.
If the data store is unreachable, the LHC contains enough information about the farm to allow normal operations for an indefinite period, if necessary. However, no new static information can be published, or added to the farm, until the farm data store is reachable and operational again.
The disappearance of LHC
With the release of the awful version 7.0 of XenApp in 2013 and the move to XenDesktop FlexCast Management Architecture (FMA), Citrix decided to remove the Local Host Cache feature–and many others–without offering any other alternative. To be fair, Citrix converged XenApp into XenDesktop, which was already using the FMA design since the version 5 and without Local Cache Host equivalent. This decision immediately made the SQL infrastructure a critical piece of any XenApp implementation. Any downtime on the SQL infrastructure would immediately cause a downtime for new sessions on the XenApp infrastructure as well. It could also have some side effects with the old Citrix Web Interface.
Citrix recommends having a highly available SQL infrastructure to host XenApp and XenDesktop databases. While you can successfully implement HA for your SQL infrastructure, it does not necessarily mean that you will avoid downtimes, as many components are to be considered.
Facing a storm of complaints, Citrix also started–finally!–to listen to its customers and released XenDesktop 7.6 in Sept 2014 with the Connection Leasing (CL) feature enabled by default.
Unfortunately, CL was not full replacement of LHC and it is alternative option provided in placement of LHC, limited to frequently used and assigned applications/desktops (up to 2 weeks by default). For users not using Citrix frequently or using pooled desktops, CL is completely useless and did not resolve anything. There are also many limitations: load management, workspace control, power actions are not supported.
Citrix came up with a milestone achievement with its new idea as part of the XenDesktop 7.12 release in Dec 2016. This time, they claimed to bring back all the Local Host Cache (LHC) features from XenApp 6.5, even adding few improvements to make it more reliable. LHC feature is offered for Cloud and On Premises implementations along Connection Leasing in 7.12, but is considered the primary mechanism to allow connection-brokering operations when database connectivity to the site database is disrupted. Surprisingly, Local Host Cache feature is disabled by default. Let us expect Citrix to enable that feature by default in the next version.
When installing XenDesktop 7.12 and up, a SQL Express instance(Local DB) will be installed locally on each Delivery Controller to store the Local Host Cache. Config Synchronizer Service (CSS) takes care of the synchronization between the remote database and the Local Host Cache (Local DB). The Secondary Brokering Service (Citrix High Availability Service) takes over from the Principal Broker when an outage is detected and does all registration and brokering operations.
There are many limitations to consider with this version of LHC
In conclusion, it took Citrix almost 4 years to deliver a somewhat equivalent of the good old Local Host Cache for XenDesktop 7.x. The database is not a single point of failure anymore in a XenDesktop/XenApp deployment. However, customers with large deployments are not supported with this version of the Local Host Cache and some of the -HUGE- limitations can discourage you from using that feature
PVS stream service abrupt termination intermittently (approx. once in month) which causing user sessions freeze and user unable to launch HSD’s.
Explained to customer that missing of best practices will lead to these type of intermittent issues , since there is no root cause found and it is not a best practice to keep cache on server in production environment , prepared a plan to change cache configuration to” Cache on RAM overflow to HDD”.
Current PVS Storage configuration for cache as below
PVS1 (VM)->1700 GB allocated through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)
PVS2 (VM) -> 1700 GB allocated through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)
Proposed Storage change Configuration as below:
Post referring multiple blogs, Write Cache proposed to all images(profiles) is 20 GB -> Therefore , for 86 HSD, 1820 GB required and it should present to complete Hyper-v cluster as HSD hosted on cluster.
How to change Cache Mode for existing production Vdisk:
To replicate to other devices(VM’s)
Since exact root cause was not found, vendor provided & self-analyzed would be the cause for abrupt streaming service termination
When each Target Device boots up the OS is not aware of the Write Cache and writes to the logical disk that it is presented (the vDisk). The PVS driver then redirects this data at the block level to the write cache which in your environment is held on the PVS server. When BNIStack driver ( transport stack for communicating with the PVS server) is then initialized, it will pull down chunks of the vDisk as and when they are needed from your vDisk store which is on your PVS server also. The BNIStack driver will also access the write cache directly when it needs to issue additional writes from the target device. The communication in relation to the above is carried out between the BNIStack driver on the target device and the Stream Service on the PVS server. The case notes state that environment have approximately 2000 users which I suspect is putting a large overhead on the Stream Service responsible for servicing all Write Cache requests.
The Target Device BNIStack driver is also responsible for retries (because UDP does not). The base timeout for a packet timeout is 10 seconds. If the server responds quickly, this value is reduced by half all the way down to 1 second. Correspondingly, if the server responds slowly the timeout will double all the way up to 10 seconds.
A retry timeout of 1 second or less may cause excessive I/O retries, leading to slow response and hanging of target devices which will then ultimately lead to a stream service failure. Since we know we are seeing issues with the stream service and subsequent hanging/crashing of this service and considering we are aware of the large write cache overheads incurred on the stream service already, I suspect that the load we are putting on the stream service is too great, culminating in it grinding to a halt, this will then present itself with the symptoms we see at the target device e.g. slow logging in, sluggishness inside the session.
As explained above, the recommendation to mitigate this is to move the Write Cache away from the PVS Server itself and place the processing overhead at the target device instead.
Recommendations & Reference
Citrix provides servicing options to give greater flexibility and choice in how to adopt new XenApp, XenDesktop, and XenServer functionality while giving greater predictability for maintaining and managing the support of your environment
Last year, Citrix introduced two new XenApp / XenDesktop servicing options, the LTSR, which stands for Long Term Service Release and the CR a.k.a. Current Release., In 2016, Citrix announced first LTSR of XenApp and XenDesktop 7.6 and in 2017 first LTSR for XenServer 7.1 that is available for download on Citrix.com.
What is LTSR?
As a benefit of Software Maintenance, Long Term Service Releases (LTSR) of XenApp ,XenDesktop,XenServer enable enterprises to retain a particular release for an extended period of time while receiving minor updates that provide fixes, typically void of new functionality. Long Term Service Releases (LTSR) is ideal for large enterprise production environments where you would prefer to retain the same base version for an extended period
A Long Term Service Release guarantees 5 years of mainstream support and an optional 5 years of extended support (needs to purchased separately). This includes cumulative updates every 4 to 6 months, a new LTSR version of XenApp / XenDesktop every 12 to 24 months and any potential (hot) fixes
A valid Software Maintenance (SM) contract is needed to make use of the LTSR or CR servicing option.
Ideal customer environment for a LTSR is for the customers who typically follow a 3-5 year version upgrade cycle
Long Term Service Releases will have a regular cadence of Cumulative Updates that will typically contain only fixes
What is Current Release?
Any new release of XenApp/XenDesktop/XenServer will be labeled a Current Release. With the CR servicing option you can always make use of (install) the most recent XenApp and/or XenDesktop versions including all the latest enhancements and additions that come with it.
Its release cycles are much shorter with a new version release being announced every three to nine months in general.
Citrix recommends that large enterprise customers have a combination of Current Release and Long Term Service Release environments.
Switching from a LTSR to a CR servicing, and vice versa, is always optional as well
All initial releases of XenApp/XenDesktop/XenServer will be a Current Release. There will likely be multiple Current Releases of a major XenApp/XenDesktop/XenServer version (i.e. 7.6, 7.6 FP1, 7.6 FP2, 7.6 FP3, 7.7, 7.8 ,7.9,7.11,7.13,7.14); however, there will likely only be one LTSR release of that version after that release is considered customer-tested and industry-proven (i.e. 7.6 FP3).
How will the customer know if their environment is Long Term Service Release compliant?
Citrix support and engineering have developed the LTSR Assistant tool which will scan your environment and compare your environment with the necessary LTSR components to determine if you are compliant. The tool provides a report that will outline the necessary updates to achieve compliance. The LTSR Assistant tool is available for download athttp://support.citrix.com/article/CTX209577.
Will a customer running an LTSR compliant environment be supported if they also have non- compliant components?
Citrix does not recommend mixing non-compliant components. For example, if a customer decides to implement Provisioning Services 7.7, which is not compliant with the current 7.6 LTSR environment and the customer has an issue with Provisioning Services 7.7 the customer may be asked to move to the latest Provisioning Services Current Release to receive public fixes
How often will Citrix release a Long Term Service Release of XenApp and XenDesktop or XenServer?
Citrix will release a Long Term Service Release of XenApp and XenDesktop or XenServer based on the number of features, implementations, customer support cases and general feedback. However, as very general guidance it can be expected that Citrix will release a new Long Term Service Release every 12-24 months; however, Citrix reserves the full rights to alter those timelines.
Is Citrix discontinuing the process of providing Hotfix Rollup Packs (HRP) for XenApp and XenDesktop?
With LTSR, Cumulative Updates will replace Hotfix Rollup Packs (HRP). Hotfix Rollup Packs (HRP) will still be made available for XenApp 6.5.
Will 7.6 LTSR support XenApp for Windows Server 2008 R2 for 10 years?
Windows Server 2008 R2 will not be eligible for extended support. Citrix will continue to monitor Windows 2008 R2 lifecycle dates for future determination of lifecycle milestones.
|No LHC||LHC(Local Host Cache)|
|Connection Leasing||No Connection Leasing|
|Pulls all information, static as well as dynamic from the central Site database||Has static as well dynamic(run-time) information cached locally|
|There is no direct communication between delivery controllers.
No scheduled communication between the VDA’s and/or Site databases, only when needed.
|Communicates with the IMA store, Peer Data Collectors and its session Hosts(within its own zone) on a scheduled interval, or when a Farm configuration change has been made.|
|Is responsible for brokering and maintaining new and existing user session only.||Often hosts user session, but can be configured as a dedicated data collector as well.|
|Can have a different operating system installed then the server and desktop VDA’s||Need to have the same operating system as all other session hosts and DC’s within same Farm.|
|Core services installed only. The HDX stack is part of VDA software||Has all the XenApp 6.5 or earlier bits and bytes fully installed.|
|Zones are optional. When configured they do need at least one Delivery controller present.||Each Zone has one Data Collector. Having multiple data collector means multiple zones.|
|Election does not apply.
Deploy multiple, at least 2 Delivery controllers per site /zone (again one per zone is the minimum)
|Can, and sometimes need to be elected. Configure at least one other Session Host per zone that can be elected as a Data Collector when needed.|
|When Central Site DB is down, no site wide configuration changes are possible. By default, Connection Leasing will kick in, enabling users to launch sessions which are assigned at least once during last 2 weeks prior to DB going offline.||when IMA Db is down, no Farm wide configuration changes are possible. Everything else continues to work as expected due to the LHC present on the Data Collectors and Session Hosts in each Zone.|
|A Delivery controller can have a direct connection(API) with a Hypervisor or cloud platform of choice.||Does not have any direct connection(API) with a Hypervisor or cloud platform management capabilities.|
|Almost all the communication directly flows through a Delivery Controller to Central Site DB.||Session Hosts as well as Data Collectors directly communicate with IMA database.|
|VDA’s needs to successfully register themselves with a Delivery controller.||When a XenApp server boots it needs a IMA service but it will not register itself anywhere.|
IMA back then…
FMA as it is today…
|IMA – Independent Management Architecture.||FMA – Flexcast Management Architecture|
|Worker Group.||Machine Catalog / Delivery Group.|
|Worker / Session Host / XenApp server.||Virtual Delivery Agent (VDA). There is a
desktop OS VDA as well as a server OS
VDA, including Linux.
|Data Collector (one per zone).||Delivery Controller (multiple per Site).|
|Zones.||Zones (as of version7.7)|
|Local Host Cache (LHC).||Conenction Leasing|
|Delivery Services Console / App center.||Citrix Studio (including StoreFront) and
|EdgeSight monitoring (optional).||Partly built into Director|
|Application folders.||Application folders (new feature in 7.6) and
Tags (all 7.x versions).
|IMA Data store.||Central Site database (SQL only).|
|Load evaluators.||Load management policies.|
|IMA protocol and service.||Virtual Delivery Agents / TCP.|
|Farm Administrators.||Delegated Site Administration using roles
and scopes, which are configurable as well.
|Citrix Receiver.||Citrix X1 Receiver. It will provide one
interface for both XenApp / XenDesktop as
well as XenMobile.
|Smart Auditor.||Session Recording.|
|Shadowing users.||Microsoft Remote Assistance, launched
|USB 2.0.||USB 3.0. Support|
|Session Pre-launch and Session Lingering.||Session Pre-launch and Session Lingering.
Both have been re-introduced.
|Power and capacity management.||Basic power management from the GUI- advanced via powershell|
|Webinterface / StoreFront.||Webinterface / StoreFront.|
|Single Sign-on for all or most of applications||There is no separate Single Sign-On
component available for XenApp 7.x. This is
now configured using a combination of StoreFront, Receiver and policies.
|Installed hotfixes inventory||Installed hotfixes inventory from studio|
|Support for Windows Server 2003 and
|FMA 7.x supports Windows Server 2008
R2 ,Server 2012 R2,2016
Issues encountered post deployment of Netscaler 10.5
Customer imported NetScaler 10.5 VPX to Hyper-v and requested us to configure further configurations
Issue 1:Netscaler URL is not opening over internet
Observations & changes done:
Netscaler has 3 Interfaces ( DMZ, LAN Zone & Loopback)
Netscaler IP’s as below
After above configurations, netscaler web page opening over internet but observed certificate errors and Authentication issue
User getting error that the credentials are incorrect when logging to Netscaler
The LDAP configuration was not as per the article http://support.citrix.com/article/CTX108876 correcting which rectified the behavior of incorrect username password.
Issue 3: Certificates errors on Netscaler.
Observations & changes done:
As per article http://support.citrix.com/article/CTX114146
VDI launching is working with internal URL and not working externally, throwing VDI error
Observations & Changes done
Using the article http://support.citrix.com/article/CTX139963
For Receiver, need to configure account services address (Similar to Xenapp Services URL)
Error: Cannot complete request, before log into Netscaler webpage and issue is same from internal URL too.
After doing all above changes, Users are able to launch VDI externally and internally without any issues
Few months back, I had an issue with one of the xen server, we had 2 servers (Server1 & Server2) in a pool. server1 is down and unable to bring it up due to hardware issue.
xe host-list params=uuid,name-label,host-metrics-live
xe vm-list is-control-domain=false resident-on=fdd15cbc-42ce-4154-8731-9a469ce04976 (Format : xe vm-list is-control-domain=false resident-on=UUID_of_failed_server)
Format : xe vm-reset-powerstate uuid=<UUID_of_the_VM_to_recover> –force