EVA 4400 Cache Battery problems - EventID 1511 and 1521

It all started so well. At 7am on my day off (!) the phone went and I was told that no-one could log in successfully. Several people were managing to get in with a temp profile but couldn't access any local apps. Luckily most of our apps are run via Citrix, which was still available via Receiver and the Start menu.

Upon inspection, lots of EventID 1511 and 1521's appearing. The advice online - rebuild the corrupt profile. Well, this was happening for the entire office of over 100 people so that was an immediate no-go.

(***In a nutshell - if you get EVENTID 1511 and 1521, it usually means a corrupted profile. In our case, the SAN drive that contained the profile folders was not presented successfully to the server, so the profile folders were not available to the users as they tried to log in.***)

We're in the process of migrating from our old EVA 4400 to the latest and greatest HP has to offer. However, being 'in the process of' means we're running both systems in parallel. The disk from the new EVA was working fine. The disk from the old EVA just shows up for 5-10 minutes after the server boots up and then disappears from view. Nothing in Disk Management either. We removed the automatic updates that installed the night before, nothing. Swapped the Fibre Card over, nothing. Checked Command View on the management system and the EVA reported full health - apart from a Cache battery on Controller 1 having died. 

The Cache battery had died on us several months before, but had no side effects on the system. This time, for some strange reason, because the cache battery on Controller 1 had died, the MPIO routing from Windows died with it. It steadfastly refused to re-route the traffic to Controller 2 and as such MPIO on the server decided that it couldn't see the drive at all. Even using CV we couldn't get the system to fail over to the working controller 2. Very frustrating. Luckily we had a spare cache battery laying around.

After swapping the cache battery, bingo - the drive reappeared! Controller 1 came back up and the system went back to normal - almost. 

However... Read the next blog for the DFS fall-out issues!