Recently I was working on a XenDesktop 7.6 project where we were seeing virtual machines continuously powering on and off occur when more than one machine catalog was added to a delivery group.
Looking at the event logs on the Delivery Controllers, we noticed Event 3013 occurring every couple of minutes (Idle Pool TurnOn and Idle Pool initiated Shutdown).
Event ID 3013-Citrix Broker Service
The Citrix Broker Service successfully performed power action ‘TurnOn’ (origin: IdlePool) on virtual machine ”Domain\Machine Name’.
Event ID 3013-Citrix Broker Service
The Citrix Broker Service successfully performed power action ‘Shutdown’ (origin: Untaint) on virtual machine ‘Domain\Machine Name’.
Our delivery groups are set to ShutDownAfterUse so this would appear to be normal at first glance, but with all the virtual machines being shut down and powered on we were starting to get a lot of unregistered machines and the amount significantly increased when users logged off during shift changes.
To see details on Power Actions being sent to VMware Virtual Center I ran Get-brokerhostingpoweraction -State ‘Pending,Started’ and saw there were 500+ pending power actions queued up.
Next step was to clear up the pending queue as fast as possible before users started calling in with issues connecting to their desktop so I increased the Hosting Connection Maximum new Actions per minute to 45 to temporarily fix the issue of unregistered virtual machines during shift change and clear out the pending actions.
I don’t recommend changing the default value unless you have the hardware and resource to be able to support it.
I first notice the issue when I logged into Virtual Center. I knew something wasn’t right when you continuously see Power On/Initiate Guest OS Shutdown initiated by the XenDesktop Hosting connection service account at continuously occurring. The screenshot only shows events from our testing. In production we saw Power On and Shutdown’s occurring almost every minute.
I still needed to find out why there were so many virtual machines turning off and on when our power policy was set to keep all virtual machines power on 24/7.
Reviewing the power policy and all the relevant Citrix documentation on idle pool management it appeared that we had everything set correctly so I decided to verify that adding more than 1 machine catalog to a delivery group caused our virtual machines to power cycle.
I started with one machine catalog with 500 virtual machines and added that to a new delivery group. Next, I adjusted the power policy to keep the virtual machines always powered on 24/7. After waiting 30 minutes I monitored virtual center tasks & events to see if any virtual machines were being shut down by the hosting connection service account. There were no Initiate Guest OS Shutdown or Power On virtual machine events for 24 hours.
After 24 hours of no power actions occurring, I created another 2 Machine Catalogs with 500 virtual machines each and added these virtual machines to the existing Delivery group with 500 virtual machines. A total of 1500 machines were added to the delivery group and the power policy was adjusted to turn on all 1500 virtual machines and keep them on 24/7.
As soon as these new machines were added we started seeing virtual machines shutting down and powering on randomly.
I also tried adjusting the default values from 10 to 100 for PeakBufferSizePercent and OffPeakBufferSizePercent.
Set-BrokerDesktopGroup “Desktop Group Name” –PeakBufferSizePercent 100
Set-BrokerDesktopGroup “Desktop Group Name” –OffPeakBufferSizePercent 100
Set-BrokerDesktopGroup “Desktop Group Name” –ShutDownDesktopAfterUse $True
Set-BrokerDesktopGroup “Desktop Group Name” –AutomaticPowerOnForAssignedDuringPeak $True
The changes above did not correct our issue and were changed back to the default values.
One other recommendation was to change the MaxRegistrationDelayMin value, but the default is 20 minutes so this was not the problem. We would have seen several virtual machines listed as Powered ON but unregistered.
After all the troubleshooting that we went through here are the 2 recommendations on how to fix this issue-
- Install the recently released Citrix Public Hotfix Update 3 on your delivery controllers which will allow you to use 1 machine catalog with more that 999 virtual machines. Before this public fix was released you would need to request the private hotfix from Citrix.#LC2873. http://support.citrix.com/article/CTX201881
- Create a machine catalog of 999 and only one delivery group of the same. Set up your Machine Catalogs and Delivery Groups so that there is always a 1 to 1 relationship. Machine Catalog A virtual machines are added to Delivery Group A, Machine Catalog B virtual machines are added to Delivery Group B.Separate the users into new security groups and add those to the new delivery groups.