EVA 4400 Cache Battery problems - EventID 1511 and 1521

It all started so well. At 7am on my day off (!) the phone went and I was told that no-one could log in successfully. Several people were managing to get in with a temp profile but couldn't access any local apps. Luckily most of our apps are run via Citrix, which was still available via Receiver and the Start menu.

Upon inspection, lots of EventID 1511 and 1521's appearing. The advice online - rebuild the corrupt profile. Well, this was happening for the entire office of over 100 people so that was an immediate no-go.

(***In a nutshell - if you get EVENTID 1511 and 1521, it usually means a corrupted profile. In our case, the SAN drive that contained the profile folders was not presented successfully to the server, so the profile folders were not available to the users as they tried to log in.***)

We're in the process of migrating from our old EVA 4400 to the latest and greatest HP has to offer. However, being 'in the process of' means we're running both systems in parallel. The disk from the new EVA was working fine. The disk from the old EVA just shows up for 5-10 minutes after the server boots up and then disappears from view. Nothing in Disk Management either. We removed the automatic updates that installed the night before, nothing. Swapped the Fibre Card over, nothing. Checked Command View on the management system and the EVA reported full health - apart from a Cache battery on Controller 1 having died. 

The Cache battery had died on us several months before, but had no side effects on the system. This time, for some strange reason, because the cache battery on Controller 1 had died, the MPIO routing from Windows died with it. It steadfastly refused to re-route the traffic to Controller 2 and as such MPIO on the server decided that it couldn't see the drive at all. Even using CV we couldn't get the system to fail over to the working controller 2. Very frustrating. Luckily we had a spare cache battery laying around.

After swapping the cache battery, bingo - the drive reappeared! Controller 1 came back up and the system went back to normal - almost. 

However... Read the next blog for the DFS fall-out issues! 

The requested operation cannot be completed because the terminal connection is currently busy processing a connect, disconnect, reset or delete operation.

I get this one regularly at work now when one of servers crash. It seems that their logon session remains connected and won't reset. There are plenty of solutions out there, but none worked for me until I found this one - so simple!
Run command prompt on the relevant server the user is having an issue with, and use the 'query process winlogon.exe'
This will show you all the winlogon processes on the server.
Find the one process without a SESSIONNAME (i.e. the only blank entry).
Make a note of the PID associated with that process.
Run Task Manager and click View menu > Select Columns... and choose PID (at the very top).
Now you can sort the list by PID number and find the winlogon process with the PID you are looking for.
Kill the winlogon process and Hey Presto, the user can now log in!

Hope this helps, took me a while to find!

Citrix... Again!

Halfway through Monday morning I get a note to say one of our users cannot access any Citrix published applications on one of our Citrix Servers (XenApp on Windows 2008). All other applications are fine, but anything on our second application server just gives the generic 'loading' message. As the day progresses, anyone else that tries to log on or create new sessions (key words: New Sessions) gets the 'loading' message. All users with open sessions are fine, programs keep working and they can be as productive as they choose.

We decide to cut our losses and bounce the server - in the past this has meant a 5 minute outage for everyone but after this all users can get onto the system. This time round however, the system won't remotely reboot. It sits there halfway through the reboot process and becomes completely unresponsive. Luckily we have a colleague down at the remote site with the server and it is forced to shut down and reboot. This does not make it happy! Over the next few hours it has to be repeatedly forced to shutdown until eventually it seems to catch up with itself and goes down gracefully. Luckily it comes back up too!

At this point, we also discovered issues with Terminal Server on the application server - it would not show any TS details and when it did load (which was infrequently) it would not show any license servers at all. We decided to reinstall TS on this server. This went without too much of a hitch but did not fix the Citrix issues.

Citrix was now showing the '*published application* failed to start' error. Little info on the web regarding this, so we ended up doing the following to register all the dll files in the webem folder in the system32 folder:
cmd> CD c:\windows\system32\webem
>for %s IN (*.dll) DO regsvr32 /s %s
Followed by:
>winmgmt /resetrepository

This allowed all the terminal server services to come back online and for us to examine and confirm that this at least was back online.

Finally, we had to make some amendments to the Registry - running regedit we found that the following key was missing some info:
HKLM\Software\Microsoft\WindowsNT\CurrentVersion\WinLogon
the AppSetup key was showing UsrLogon.cmd
but needed to show:
CtxHide.exe UsrLogon.cmd,cmstart.exe

Once these changes had been made, Citrix was back up and running again! Now to do some clean-up routines and find out what exactly went wrong in the first place!