Ram's Virtual Tech Site

All about Virtualization -Personal Blog

Category: Xen Desktop

PVS Streaming Service Abrupt Termination – Cache Change Mode Procedure for production vdisk

Issue:

PVS stream service abrupt termination  intermittently (approx. once in month) which causing user sessions freeze and user unable to launch HSD’s.

Environment :

2 Citrix PVS Servers (VM’s) with version 7.6
2000-3000 concurrent  Users
86 HSD’s & 6 Golden Images
Microsoft Hypervisor 2012R2 ( 15 Node) – CICSO UCS

Observations:

  • Issue occurring once or twice in a month and there is no common pattern in days or hours,issue recurring in both PVS servers at a time
  • No changes in environment
  • Onsite engineer informed that issue existed since 3 months and issue getting resolved post restart of PVS servers.
  • One day,  same issue repeated but issue not sorted out post restarting of PVS servers -> Issue escalated to support team (Me)
  • Observed  Event Id 11 :”Detected one or more hung threads , DbAccess error: <Record was not found> <-31754> (in ServerStatusSetDeviceCount() called from SSProtocolLogin.cpp:2903” -> Indicates “Thread hangs under the stream service” & DB Access errors
  • Observed multiple vDisk retries on the problematic target devices. 11 at boot time and approximately 611 per hour during session
  • Observed recommended MacAfee exclusions are not in place -> Stopped MacAfee service and restarted PVS server -> PVS Streaming service stable for some time on one PVS server  and again terminated ->Due to time constraint, logged a call with vendor(Citrix).
  • After 2 hrs, Citrix support joined the call and started collecting CDF races and procdump collection for the terminating stream service
  • After few hours , issue resolved automatically and Citrix support unable to find root cause with collected logs
  • In 2 months , issue repeated 2 times and customer frustrated as root cause was not found for abrupt streaming service termination intermittently.
  • Support Team (Myself) analyzed the environment and observed the Cache mode is configured as “ Cache on Server”  which is not recommended for Production environment , Best practice to use “Cache on RAM overflow to HDD”  which is a best practice to reduce load on PVS server & optimal performance ->Taken the same observation Citrix support and requested their observations

Explained to customer that missing of best practices will lead to these type of intermittent issues , since  there is no root cause found  and it is not a best practice to keep cache on server in production environment , prepared a plan to change cache configuration to” Cache on RAM overflow to HDD”.

Current PVS Storage configuration for cache as below

PVS1 (VM)->1700 GB allocated  through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)

PVS2 (VM) -> 1700 GB allocated through Virtual HBA ( Total golden Image Sizes is 440 Gb & Remaining for Write Cache)

Proposed Storage change Configuration as below:

Post referring multiple blogs, Write Cache proposed to all images(profiles) is 20 GB -> Therefore , for 86 HSD, 1820 GB required and it should present to complete Hyper-v cluster as HSD hosted on cluster.

XenDesktop Controller Hotfix Update Procedure

Implementation Plan

  • Take a Full backup of Citrix Databases on server locally and tapes.
  • Take a snapshot of DDC01 (Controller 1)
  • Download and Install Hotfix update 1(CTX135207) on DDC01(Controller 1)
  • Reboot DDC01
  • Test VDI by stopping the services in DDC02 so that session will be established to DDC01.
  • Take a snapshot of DDC02 (Controller 2)
  • Install Hotfix update 1 on DDC02(Controller 2) – Similar procedure of DDC01
  • Reboot DDC02
  • Test VDI by stopping the services in DDC01 so that session will be established to DDC02
  • Observe for 1 week and remove snapshots.

Roll Back

  • Uninstall the component from ARP/Programs and Features.
  • Restore the data store as described in Knowledge Center article CTX135207.
  • Install the desired level of the component (base or other hotfix).
  • Restart the Controller even if not prompted to do so

OR

Revert the snapshot which was taken before installation

Find the document with screenshot in attachment

Document – Xen Desktop7.5 Hotfix Update Installation Procedure

Additional Information

Anitivirus Exclusion List – Hyper-v/Citrix

The following antivirus exclusions should be applied to all Citrix infrastructure servers:

  • Set real-time scanning to scan local drives only and not network drives
  • Disable scan on boot
  • Remove any unnecessary antivirus related entries from the Run key
  • Exclude the pagefile(s) from being scanned
  • Exclude IIS log files from being scanned
  • Exclude Windows event logs from being scanned

Hyper-V

Exception Item Description
VHD,VSV,ISO,AVHD,VFD,XML,BIN,VHDX,AVHDX,HRL Extensions
C:\ProgramData\Microsoft\Windows\Hyper-V\*.* Directory & Subdirectories
C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks\*.* Directory & Subdirectories
C:\ProgramData\Microsoft\Windows\Hyper-V\Snapshots Directory
C:\ClusterStorage\*.* Directory & Subdirectories
Vmms.exe Process exclusions
Vmwp.exe Process exclusions
Clussvc.exe Process exclusions

 

Citrix (XenApp/Xen Desktop)

 

Component Exclusion List
Citrix Director & StoreFront Director and StoreFront:
\inetpub\temp\IIS Temporary Compressed Files
\Windows\system32\inetsrv\w3wp.exe
\Windows\SysWOW64\inetsrv\w3wp.exe StoreFront:
\Program Files\Citrix\Receiver StoreFront\Services\SubscriptionsStoreService
Citrix Profile Manager Agent:
Do not scan on open or status-check operations
UserProfileManager.exe
EdgeSight Agent:
<AllUsersProfile>\Application Data\Citrix\System Monitoring\Data
\ProgramFiles\Citrix\System Monitoring\Agent\Core\rscorsvc.exe
\ProgramFiles\Citrix\System Monitoring\Agent\Core\Firebird\bin\fbserver.exe Server:
\CommonProgramFiles\Citrix\System Monitoring\Server\RSSH
\ProgramFiles\Citrix\System Monitoring\Server\EdgeSight\scripts\rssh
\ProgramFiles\Citrix\System Monitoring\Server\EdgeSight\Pages
\ProgramFiles\Microsoft SQL Server\MSSQL\Reporting Services
\ProgramFiles\Microsoft SQL Server\MSSQL\Data
\SystemRoot\SYSTEM32\Logfiles
Provisioning Services – For Server
Note: An even easier approach would be to
 exclude the complete Provisioning services folder
For PVS Server
\Windows\System32\drivers\CvhdBusP6.sys (Windows Server 2008)
\Windows\System32\drivers\CVhdMp.sys (Windows Server 2012)
\Windows\System32\drivers\CfsDep2.sys
\Program Files\Citrix\Provisioning Services\BNTFTP.EXE
\ProgramData\Citrix\Provisioning Services\Tftpboot\ARDBP32.BIN
\Program Files\Citrix\Provisioning Services\StreamService.exe
\Program Files\Citrix\Provisioning Services\StreamProcess.exe
\Program Files\Citrix\Provisioning Services\soapserver.exe
C:\Windows\System32\drivers\CVhdBusP6.sys => (PVS 6.1)
C:\Windows\System32\drivers\CVhdBus2.sys => (PVS 5.6)
C:\Windows\System32\drivers\CFsDep2.sys => (PVS 5.6 and PVS 6.1)
C:\Program Files\Citrix\Provisioning Services\BNTFTP.EXE => (PVS 5.6 and PVS 6.1)
C:\ProgramData\Citrix\Provisioning Services\Tftpboot\ARDBP32.BIN => (PVS 5.6 and PVS 6.1)
D:\Store => ( i.e. local vdisk store) ->Exclude scanning of Local vDisk Store
Provisioning Services – For Target Devices
Note: An even easier approach would be to
 exclude the complete Provisioning services folder
Target Devices:
\Program Files\Citrix\Provisioning Services\BNDevice.exe
\Windows\System32\drivers\bnistack6.sys
\Program Files\Citrix\Provisioning Services\TargetOSOptimizer.exe
\Windows\System32\drivers\CfsDep2.sys
\Windows\System32\drivers\CVhdBusP6.sys
\Program Files\Citrix\Personal vDisk\BIN\WIN7\
C:\Windows\System32\drivers\bnistack.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\bnistack6.sys => (Only targets, 2008/Win7)
C:\Windows\System32\drivers\BNNF.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\BNNS.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\BNNS6.sys => (Doesn’t exist anymore with PVS6.1 Agent)
C:\Windows\System32\drivers\BNPort.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\CFsDep2.sys => (Win2003/XP & 2008/Win7)
C:\Windows\System32\drivers\CVhdBusP52.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\CVhdBusP6.sys => (2008/Win7)
C:\Program Files\Citrix\Provisioning Services\BNDevice.exe => (Only targets, 2008/Win7)
C:\Program Files\Citrix\Provisioning Services\TargetOSOptimizer.exe => (Only targets, 2008/Win7)
Target – Personal vDisk:
CTXPVD.exe
CTXPVDSVC.exe
Exclude scanning of Write Cache
Xen App -Session Controller(Controller) Controller:

\Windows\system32\csrss.exe
\Windows\system32\winlogon.exe
\Windows\system32\userinit.exe
\Windows\system32\smss.exe
\Program Files\Citrix\Group Policy\Client-Side Extension\CitrixCseEngine.exe
\Program Files (x86)\Citrix\System32\wfshell.exe
\Program Files (x86)\Citrix\system32\ctxxmlss.exe
\Program Files (x86)\Citrix\System32\CtxSvcHost.exe
\Program Files (x86)\Citrix\system32\mfcom.exe
\Program Files (x86)\Citrix\System32\Citrix\Ima\ImaSrv.exe
\Program Files (x86)\Citrix\System32\Citrix\Ima\IMAAdvanceSrv.exe
\Program Files (x86)\Citrix\HealthMon\HCAService.exe
\Program Files (x86)\Citrix\Streaming Client\RadeSvc.exe
\Program Files (x86)\Citrix\Streaming Client\RadeHlprSvc.exe
\Program Files (x86)\Citrix\Independent Management Architecture\RadeOffline.mdb
\Program Files (x86)\Citrix\Independent Management Architecture\imalhc.mdb

Xen App -Session Host Session Host:

\Windows\system32\spoolsv.exe
\Windows\system32\csrss.exe
\Windows\system32\winlogon.exe
\Windows\system32\userinit.exe
\Windows\system32\smss.exe
\Program Files\Citrix\Group Policy\Client-Side Extension\CitrixCseEngine.exe
\Program Files (x86)\Citrix\System32\wfshell.exe
\Program Files (x86)\Citrix\system32\CpSvc.exe
\Program Files (x86)\Citrix\System32\CtxSvcHost.exe
\Program Files (x86)\Citrix\system32\mfcom.exe
\Program Files (x86)\Citrix\System32\Citrix\Ima\ImaSrv.exe
\Program Files (x86)\Citrix\System32\Citrix\Ima\IMAAdvanceSrv.exe
\Program Files (x86)\Citrix\HealthMon\HCAService.exe
\Program Files (x86)\Citrix\Streaming Client\RadeSvc.exe
\Program Files (x86)\Citrix\Streaming Client\RadeHlprSvc.exe
\Program Files (x86)\Citrix\XTE\bin\XTE.exe
\Program Files (x86)\Citrix\Independent Management Architecture\RadeOffline.mdb
%AppData%\ICAClient\Cache (if using pass-through authentication)

XenDesktop – Controller Controller:

\Windows\system32\csrss.exe
\Windows\system32\winlogon.exe
\Windows\system32\userinit.exe
\Windows\system32\smss.exe

Controller – pre-XenDesktop 7.x:

\Program Files\Citrix\Group Policy\Client-Side Extension\CitrixCseEngine.exe
\Program Files (x86)\Citrix\System32\wfshell.exe
\Program Files (x86)\Citrix\system32\ctxxmlss.exe
\Program Files (x86)\Citrix\System32\CtxSvcHost.exe
\Program Files (x86)\Citrix\system32\mfcom.exe

Windows Server OS Machines – XenDesktop 7.x:

\Windows\system32\spoolsv.exe
\Windows\system32\csrss.exe
\Windows\system32\winlogon.exe
\Windows\system32\userinit.exe
\Windows\system32\smss.exe
\Program Files\Citrix\Group Policy\Client-Side Extension\CitrixCseEngine.exe
\Program Files (x86)\Citrix\System32\wfshell.exe
\Program Files (x86)\Citrix\system32\CpSvc.exe
\Program Files (x86)\Citrix\System32\CtxSvcHost.exe

Ref:

 

Hotfix Name Changes for XenApp/XenDesktop 7.5

Information

This article explains the changes for Citrix hotfix naming conventions in XenDesktop 7.1/7.5 with the introduction of XenApp 7.5.

 

Hotfix Name Changes

 

With the reintroduction of XenApp in version 7.5, the same underlying components are used for XenApp and XenDesktop. As a result, the ‘XA’ and ‘XD’ designation will not appear in the hotfix name. Instead, the component name is prepended to the hotfix name.

 

Example 1

A hotfix previously named XD750DStudioWX86001 will now be DStudio750WX86001.

 

Hotfix Version Number Association

The following components did not change between version 7.1 and 7.5. From now on, updates to these components will only contain the 7.5 association in the name. The hotfixes will be available and compatible with both the 7.1 and 7.5 component versions.

  • Broker Agent
  • Desktop OS VDA
  • Director VDA Plugin
  • Enhanced Desktop Experience
  • Personalization AppV – Studio
  • Personalization AppV – VDA
  • Server OS VDA
  • StoreFront Privilege Service
  • Universal Print Client
  • Universal Print Server
  • WMI Proxy Plugin

 

Example 2

A hotfix previously named XD710ICAWSWX86006 will now be ICAWS750WX86006.

 

VDA Core Services Hotfixes and Machine Type Association

For the VDA core services hotfixes, the OS type is designated in the hotfix name. ‘TS’ for Terminal Server, the hotfix would apply to a Windows Server operating system. ‘WS’ for workstation, the hotfix would apply to a desktop Windows operating system. There is no correlation between the hotfix numbering for the Server OS (TS) and Desktop OS (WS) hotfixes. The ICATS hotfix ending in 007 might not have the same fixed issues as the ICAWS hotfix ending in 007.

 

Example 3

A hotfix named ICAWS750WX86007 is a VDA core services hotfix for a Windows 7, 8, or 8.1 operating system (32-bit).

A hotfix named ICATS750WX64007 is a VDA core services hotfix for a Windows Server 2008R2 , 2012, or 2012R2 operating system (64-bit).

 

Note: The previously released version 7.1 hotfixes will not be rebuilt with the new version identification, but the hotfix readme documents will reflect their support for versions 7.1 and 7.5.

 

Ref: http://support.citrix.com/article/CTX200156

Error: You cannot access this session because no licenses are available -Xen Desktop

Symptoms or Error

  • A XenDesktop session fails to start, with the following error: “You cannot access this session because no licenses are available.” Event ID: 9027, Event ID: 1163
  • Event Source: Citrix ICA Service
  • Error 0 received while obtaining a license for a Citrix XenApp client connection.
  • The license request has been rejected.
    • Note: The license error “flex code -18” shown in the DDC log stands for “License server system does not support this feature”.
  • The Broker log might contain the following errors:
    • Controller:EventLogManager decided to log event CDS_EVENT_LICENSE_NONE_CHECKED_OUT of type Warning with arguments:
      • This is based on event log groups LicensingCheckout
    • Licensing:MFLic_GetLicense result Success, request result Rejected
    • Licensing:License request rejected, flex code -18

Solution

Resolution 1 – Event ID: 9027

Check the configuration of the site: XenDesktop License edition and model using PowerShell command “Get-BrokerSite”  which shows license configuration for the site.

For Virtual Machine hosted applications

  • AppLicenseEdition
  • ApplicationLicenseModel

For XenDesktop sessions

  • DesktopLicenseEdition
  • DesktopLicenseModel

 

Compare the configuration with the licenses used in the environment from license server.  Check the licenses in the license server.  Change the site configuration using the Set-BrokerSite PowerShell command.

Examples

To configure the site to use the Platinum edition, use the following command:

  • Set-BrokerSite -DesktopLicenseEdition PLT

To set up Virtual Machine hosted applications to use Platinum edition:

  • Set-BrokerSite -AppLicenseEdition PLT

 Resolution 2:

  1. Download and install the most recent version of the licensing server.
  2. On the DDC, restart the Citrix services: (This can be done on a live system and will not affect the users)
    • Citrix AD Identity Service
    • Citrix Broker Service
    • Citrix Configuration Service
    • Citrix Diagnostic Facility COM Service
    • Citrix Host Service
    • Citrix Machine Creation Service
    • Citrix Machine Identity Service
  3. On the License Server, restart Citrix services.
    • Citrix Licensing
    • Citrix Licensing Config Service
    • Citrix Licensing Support Service
  4. Access Desktop Studio > Configuration > Licensing > Change Licensing Server > Verify.
  5. If the issue persists, restart all the DDCs in the farm one by one.

Resolution 3 – Event ID: 1163

In this case, upgrade the database as prompted by XenDesktop Studio.

Restart the Desktop Studio Server, error should no longer appear.

Resolution 4

  1. Confirm all licenses are visible in License admin console.
  2. Verify end date of subscription advantage of licenses are relevant to version of product.
  3. Confirm start up server is showing in License admin console and hosts are communicating.
  4. Import startup.lic or reinstall license admin console, which should create a new startup.lic.
    1. Issue occurs mainly due to either corrupt startup.lic or no startup.lic under C:\Program Files\Citrix\Licensing\MyFiles
  5. Restart Citrix licensing service.
  6. Relaunch application, error should no longer appear.
 Problem Cause
  • Event ID: 9027 Licensing is not set up properly in XenDesktop Site; licenses are not checked out on the license server when the user tries to start a new XenDesktop session.
  • License Model: Could be User/Device or Concurrent License Edition: Could be Platinum (PLT), Enterprise (ENT) or Advanced (ADV).
  • Event ID: 1163 XenDesktop Upgrade did not complete successfully and the following error was observed in the XenDesktop Studio: “A database upgrade is available. Learn more about this upgrade”
  • Startup license file is missing from the My Files folder.
  • corrupt startup.lic or no startup.lic under C:\Program Files\Citrix\Licensing\MyFiles

 Additional Resources

http://support.citrix.com/article/CTX136266

Find master image used by machine catalog

To load Citrix snappins ,  you need to add the Citrix powershell snapins. To do so, type : Add-PSSnapin citrix*  or asnp citrix*

Get-ProvScheme is the powershell command provides the details of provisioning scheme & Master Image

Note: The above details can also be fetched by logging to SQL configuration database(Table: DesktopUpdateManagerSchema.ProvisioningScheme))

Syntax:
MasterImageVM : XDHyp:\HostingUnits\<Hosting Resource Connection Name>\<Master Image Name>.vm\Citrix_XD_<Catalogue Name>.snapshot

ProvScheme

Note:
  • Above POST created based on personal experience & knowledge for personal reuse.
  • In case, if you wish you to use the above article, use the above steps upon proper testing and reader will be responsible for any outcome.
error: Content is protected !!