Citrix... Again!

Halfway through Monday morning I get a note to say one of our users cannot access any Citrix published applications on one of our Citrix Servers (XenApp on Windows 2008). All other applications are fine, but anything on our second application server just gives the generic 'loading' message. As the day progresses, anyone else that tries to log on or create new sessions (key words: New Sessions) gets the 'loading' message. All users with open sessions are fine, programs keep working and they can be as productive as they choose.

We decide to cut our losses and bounce the server - in the past this has meant a 5 minute outage for everyone but after this all users can get onto the system. This time round however, the system won't remotely reboot. It sits there halfway through the reboot process and becomes completely unresponsive. Luckily we have a colleague down at the remote site with the server and it is forced to shut down and reboot. This does not make it happy! Over the next few hours it has to be repeatedly forced to shutdown until eventually it seems to catch up with itself and goes down gracefully. Luckily it comes back up too!

At this point, we also discovered issues with Terminal Server on the application server - it would not show any TS details and when it did load (which was infrequently) it would not show any license servers at all. We decided to reinstall TS on this server. This went without too much of a hitch but did not fix the Citrix issues.

Citrix was now showing the '*published application* failed to start' error. Little info on the web regarding this, so we ended up doing the following to register all the dll files in the webem folder in the system32 folder:
cmd> CD c:\windows\system32\webem
>for %s IN (*.dll) DO regsvr32 /s %s
Followed by:
>winmgmt /resetrepository

This allowed all the terminal server services to come back online and for us to examine and confirm that this at least was back online.

Finally, we had to make some amendments to the Registry - running regedit we found that the following key was missing some info:
HKLM\Software\Microsoft\WindowsNT\CurrentVersion\WinLogon
the AppSetup key was showing UsrLogon.cmd
but needed to show:
CtxHide.exe UsrLogon.cmd,cmstart.exe

Once these changes had been made, Citrix was back up and running again! Now to do some clean-up routines and find out what exactly went wrong in the first place!